ANALYSIS  OF  OUTCOME-BASED 
QUALITY  ASSURANCE  SYSTEMS 


TASK  A  REPORT 


Submitted  to:   Health  Care  Financing  Administration 

Submitted  by:  Human  Services  Research  Institute 

Prepared  by:  Human  Services  Research  Institute 

Temple  University  Institute  on  Disabilities 
Jaskulski  and  Associates 


REPORTS 


RC 

570 

.5 

U6 

A53 

1996 


Analysis  of  Outcome-Based 
Quality  Assurance  Systems 


Elizabeth  Couchoud,  Project  Officer 
Health  Care  Financing  Administration 
Health  Standards  and  Quality  Bureau 
7500  Security  Boulevard 
Baltimore,  MD  21214 
(Contract  Number:  500-94-0074) 


Human  Services  Research  Institute 
2336  Massachusetts  Avenue 
Cambridge,  MA  02140 


TASK  A  REPORT 


Submitted  to: 


Submitted  by: 


Prepared  by: 


Human  Services  Research 

Institute 

Valerie  J.  Bradley 
John  Ashbaugh 
Lynda  Brown 
Madeleine  Kimmich 
Virginia  Mulkern 
Barbara  Raab 
Julie  Silver 
Marianne  Taylor 


Temple  University  Institute  on 


Jaskulski  and  Associates 

Tecla  Jaskulski 


Disabilities 

Celia  Feinstein 
Robin  Levine 
Jim  Lemanowicz 
Kathy  Miller 


July  1, 1996 


Report  #1 
State  Quality  Assurance  Systems 

ANALYSIS  OF  OUTCOME-BASED  QUALITY 
ASSURANCE  SYSTEMS 


Report  #2 

Analysis  of  the  Application  of  the  Active  Treatment  Definition 

(Objective  5,  Task  A) 


REVIEW  OF  THE  CURRENT  DEFINITION  OF  ACTIVE 
TREATMENT  AND  RELATED  ICF/MR  QUALITY 
ASSURANCE  ACTIVITIES 


Report  #1 
State  Quality  Assurance  Systems 

ANALYSIS  OF  OUTCOME-BASED  QUALITY 
ASSURANCE  SYSTEMS 


REPORT  #1 
STATE  QUALITY  ASSURANCE  SYSTEMS 
TABLE  OF  CONTENTS 


I.  INTRODUCTION  AND  RATIONALE  1 

II.  METHODOLOGY  5 

Objective  1:  To  summarize  the  "state  of  the  States"  in  outcome-based  quality 
assurance  for  ICF/MR  facilities  and  for  facilities/residences  serving 
comparable  individuals  6 

Objective  2:  To  conduct  an  in-depth  review  and  analysis  of  such  systems  in 
6-8  States  6 

Objective  3:  To  assess  the  reliability  and  validity  of  the  selected  quality 
assurance  systems   9 

Objective  4:  To  identify  and  review  empirical  studies  of  outcome-focused 
quality  assurance  systems   21 

Objective  5:  To  analyze  application  of  the  current  ICF/MR  active  treatment 
definition  and  its  consistency  with  current  concepts  of  support  26 

Objective  6:  To  identify  implications  for  changes  in  the  current  ICF/MR 
quality  assurance  system  26 

III.  RESULTS  AND  FINDINGS  27 

Objective  1:  Summary  of  Outcome  Focused-Quality  Assurance  Approaches  27 

The  Degree  Of  Quality  Assurance  Reform  27 

Category  I:  Significant  movement  toward  outcome-focused  quality 
assurance:  (10  States)  27 

Transition  II  to  I:  States  actively  in  transition  to  an  outcome-focused 
system:  (12  States)  27 

Category  II:  Some  reform  aimed  at  outcome-focused  quality 
assurance  including  the  primary  outcomes  of  interest:  (9  States)  28 

Transition  III  to  II:  Beginning  the  process  of  developing  some  aspects 
of  an  outcome-focused  quality  assurance  system:  (5  States)  28 

i 


Category  III:  Little  or  no  movement  toward  outcome-focused  quality 


assurance  reforms:  (12  States)  28 

Other:  (3  States)  28 

Scope  Of  Facilities /Residential  Services  And  Populations  Covered  31 

Selection  Of  Quality  Outcomes  35 

Objective  2:  State  Approaches  to  Outcome-Focused  Quality  Assurance  in 
Selected  States  '.  37 

Overview  37 

State  Systems  37 

Service  System  Configuration  38 

Quality  Assurance  Capability  40 

Primary  Outcome-Focused  QA  Mechanisms  40 

The  States'  Shift  To  Outcomes-Focused  QA  45 

Implementation  Challenges  45 

Aspects  Of  Outcomes-Focused  QA  46 

Applicability  To  Various  Settings  and  Various  Populations  47 

Themes  and  Recommendations  50 

Objective  3:  Review  of  the  reliability  and  validity  of  selected  quality 
assurance  instruments  56 

Summary  of  Findings  >.56 

Implications  and  Recommendations  59 

Objective  4:  Results  of  the  review  of  empirical  studies  of  outcome  focused 
quality  assurance  60 

Summary  60 

IV.  BRIEF  SUMMARY  OF  FINDINGS  AND  RECOMMENDATIONS  61 

Summary  61 

Recommendations  62 

ii 


Appendix  A 


Knowledgeable  Individuals  on  ICF/MR  Quality  Assurance  who  Provided 
Information  in  Development  of  Task  A  Interim  Report 


Appendix  B 


1  Behavioral  Development  Survey 

2  Health  and  Safety  Issues  Instrument 


Appendix  C 


1  State  by  State  Analyses  of  Psychometric  Properties  of 
Outcome-Based  Quality  Assurance  Instruments 

2  Analysis  of  Accreditation  Council  Survey 

3  Analysis  of  Commission  on  the  Accreditation  of  Rehabilitation 
Facilities  Surveys 


Appendix  D  1        Comments  to  HCFA  on  Draft  Revisions  to  the  Survey  Process  for 

ICF/MR 

2        Comments  on  Revised  Conditions  of  Participation  for  ICFs/MR 


Appendix  E 


Cumulative  List  of  all  Interviewees 


Appendix  F  Combined  References  (Tasks  A  &  B) 


iii 


Report  #1 
State  Quality  Assurance  Systems 

ANALYSIS  OF  OUTCOME-BASED  QUALITY  ASSURANCE 

SYSTEMS 


I.  INTRODUCTION  AND  RATIONALE 

The  following  report,  funded  by  the  Health  Care  Financing  Administration,  provides 
an  overview  of  significant  changes  taking  place  in  the  field  of  quality  assurance  in 
mental  retardation  and  developmental  disabilities,  analyzes  the  qualitative  and 
quantitative  properties  of  these  changes,  and  contrasts  these  emerging  trends  with 
current  federal  regulatory  practices. 

The  increasing  emphasis  on  outcomes  as  a  mode  for  monitoring  the  quality  of  services 
to  people  with  mental  retardation  and  other  developmental  disabilities  is  more  than 
merely  a  change  in  measurement  approaches.  It  represents  a  change  in  the  expectations 
regarding  what  is  a  "quality"  service  and,  as  such,  is  part  of  a  much  larger  shift  in  the 
field;  specifically  from  program-oriented,  formulaic  models  of  care  to  individually 
tailored  supports  based  on  individual  choices  and  preferences.  As  a  result,  the  infusion 
of  outcome-oriented  quality  assurance  techniques  is  both  the  result  of  the  change  in 
expectations  as  well  as  a  method  for  expediting  the  shift  at  the  provider  level.  There 
are  a  variety  of  reasons  why  this  shift  has  taken  place: 

•  The  expectations  of  people  with  disabilities  and  their  families  regarding  the 
quality  of  services  has  changed  as  fewer  people  have  been  placed  in 
institutions,  as  more  people  are  included  in  their  local  service  systems,  and  as 
policies  regarding  empowerment  of  people  with  disabilities  and  their  families 
have  become  more  widespread  (e.g.,  expansion  of  family  support,  the  adoption 
of  person-centered  planning,  etc.) 

•  Quality  assurance  mechanisms  have  evolved  as  the  system  has  evolved  and 
become  more  competent:  input  approaches  were  needed  when  the  major  goal 
of  public  policy  was  provide  for  people's  basic  need;  process  measures  were 
needed  when  the  emerging  technologies  and  interventions  needed  to  be 
codified  and  expanded;  and  outcome  measures  are  needed  now  to  ensure  that 
those  technologies  are  in  fact  resulting  in  an  improved  quality  of  life  for  those 
being  served  and  supported. 


•  In  order  to  provide  individually  tailored  supports,  it  is  necessary  to  allow  a 
certain  amount  of  flexibility  for  innovation  and  creativity.  Certain  process 
requirements  are  seen  by  some  providers  and  policy  makers  as  constraining 
innovation  whereas  outcome  measurements  allow  for  more  flexibility. 

•  Quality  assurance  systems  that  rely  on  punishment  and  sanctions  were 
probably  suited  to  a  period  when  the  state  of  the  art  was  not  broadly 
understood  and  when  the  basic  concerns  had  to  do  with  people's  physical  well- 
being.  They  are  also  more  consistent  with  periods  of  time  when  there  is  a 
shared  orthodoxy,  a  standardization  of  service  models,  and  a  more  hierarchical 
administrative  structure  for  providing  services.  The  emergence  of  the  supports 
model  moves  away  from  standardized  methods  of  providing  services  and 
requires  a  less  standardized  and  therefore  less  sanction  oriented  method  for 
assessing  quality. 

•  Finally,  and  linked  to  the  point  above,  the  growth  of  continuous  quality 
improvement  and  total  quality  management  initiatives  has  strongly  influenced 
the  tenor  or  quality  assurance  and  has  shifted  the  focus  of  QA  from  process 
oriented  measures  to  customer  satisfaction,  and  from  deficit  spotting  to  quality 
enhancement. 

Thus  the  larger  context  for  this  project  is  not  simply  an  analysis  of  new  techniques  but 
also  the  context  of  the  larger  reform  within  which  they  are  taking  places.  Some  of  the 
forces  at  work  in  the  states  and  nationally  include  the  following: 

•  Reduction  in  number  of  ICF/MR  beds  A  general  observation  of  the  states 
visited  is  the  rapidity  with  which  some  states  are  reducing  the  number  of 
ICF/MR  beds  in  the  public  as  well  as  the  private  sector.  In  Massachusetts,  all 
community  ICF/MR  beds  have  been  converted  to  waiver  status.  In  Colorado, 
there  are  only  300  beds  left  in  the  whole  state.  This  reduction  raises  the 
question  of  the  continuing  relevance  of  a  separate  ICF/MR  regulatory 
framework  and  the  possibility  of  a  combined  waiver/ICF/MR  quality 
assurance  approach.  As  more  and  more  Medicaid-funded  residences  are  under 
the  waiver,  there  were  also  concerns  raised  in  some  states  about  the  adequacy 
of  waiver  quality  assurance  approaches.  In  New  York,  for  example,  the  state 
was  in  the  process  of  reorganizing  and  recentralizing  quality  assurance 
monitoring  for  waiver  facilities  given  what  were  said  to  be  some  examples  of 
inadequate  oversight. 

•  Impact  of  Resource  Cut-Backs  In  many  of  the  states  visited,  cutbacks  in 
central  office  staff  had  already  taken  place  and/or  were  planned  for  the  future. 
Therefore,  it  is  clear  that  any  labor  intensive,  highly  systematic  approach  to  the 
collection  of  outcome  and  quality  assurance  information  will  be  difficult  if  not 
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impossible  to  implement  in  those  states  (e.g.,  CO)  where  the  central  office  staff 
is  diminishing.  Though  not  all  of  the  states  visited  had  undergone  such  cut- 
backs, it  is  likely  -  given  the  changes  taking  place  in  federal  funding  —  that 
there  will  be  cutbacks  in  other  states  such  as  New  York  and  Massachusetts. 
This  suggests  that  approaches  such  as  the  QUEST  in  Massachusetts  will  be  hard 
to  transfer  to  small  central  office  states. 

It  is  also  interesting  to  note  that  many  of  those  states  that  had  more  labor- 
intensive  approaches  to  quality  assurance  (e.g.,  Oklahoma,  Pennsylvania, 
Massachusetts,  and  Oregon)  were  under  a  variety  of  federal  court  mandates 
that  required  the  installation  of  systematic  quality  assurance  measures. 

•  Reduction  in  Case  Management  Resources  Again,  the  cut-back  in  resources  at 
the  state  level  is  having  an  impact  on  the  presence  of  case  managers  and  the 
numbers  of  people  on  their  caseloads.  Given  that  case  managers  are  sometimes 
viewed  as  the  front-line  of  a  quality  assurance  system  and,  in  some  instances, 
the  collectors  of  QA  information,  this  diminution  of  capacity  should  be  noted. 
The  roles  of  case  managers  in  many  states  are  also  changing  as  states  like  New 
Hampshire  are  moving  to  a  service  broker  or  support  person  approach  in 
which  the  staff  person  acts  a  collaborator  with  the  person  with  a  disability  and 
the  family  to  secure  specialized  as  well  as  generic  supports.  Thus  while 
conventional  case  management  functions  may  be  decreasing,  the  new  roles  for 
service  brokers  may  mean  that  will  be  even  better  informants  regarding 
outcomes  for  particular  individuals. 

•  Impact  of  Decentralization  Many  states  are  moving  to  significantly 
decentralize  responsibility  and  authority  for  the  provision  of  services  to  people 
with  mental  retardation  and  other  developmental  disabilities.  In  Colorado,  for 
instance  there  is  a  proposal  to  decentralize  the  system  to  the  community- 
centered  board  which  would  function  as  a  mini  "managed  care"  entity.  While 
the  state  will  likely  maintain  some  quality  assurance  responsibility,  other  QA 
functions  will  undoubtedly  be  moved  to  the  community  level.  This  devolution 
of  responsibility  raises  questions  regarding  the  roles  that  will  be  played  by  the 
state  versus  local  entities  and  the  nature  of  the  level  at  which  accountability  for 
service  quality  and  outcomes  will  be  lodged. 

•  Integration  of  Services  While  it  is  only  an  emerging  trend,  the  move  to 
combine  services  across  disabilities  -  especially  at  the  state  level,  is  beginning 
to  take  hold  in  some  states.  In  Massachusetts,  there  is  a  proposal  to  merge 
mental  retardation  services  with  services  to  people  who  are  deaf,  and  to  people 
who  are  blind.  In  New  Hampshire,  there  is  a  proposal  to  combine  services  to 
people  with  developmental  disabilities  with  services  to  the  elderly.  In  Georgia, 
the  state  has  consolidated  mental  health,  mental  retardation,  and  substance 


3 


abuse  services  under  a  regional  administration  and  is  currently  developing 
unified  performance  indicators  across  the  three  disabilities.  This  move  to 
integrate  services,  based  in  part  on  cost  concerns,  may  result  in  more  generic 
quality  assurance  measures  over  time. 

•  Relevance  of  Managed  Care  While  there  are  currently  no  managed  care 
waivers  approved  for  managed  care  in  long-term  care  for  people  with  mental 
retardation  and  other  developmental  disabilities,  there  are  plans  in  virtually 
every  state  to  develop  waiver  proposals.  The  interjection  of  managed  care  has 
profound  implications  for  the  role  of  the  federal  and  state  government  in 
quality  assurance  and  such  considerations  should  be  part  of  any  reconstituted 
federal  quality  assurance  guidelines.  Specifically,  some  data  collection  and 
oversight  functions  may  be  delegated  to  private  managed  care  companies.  In 
Georgia,  for  instance,  the  state  is  currently  contemplating  requiring  managed 
care  companies  to  develop  their  own  accreditation  mechanisms.  The  advent  of 
managed  care  may  also  further  accelerate  the  decline  in  the  utilization  of  more 
expensive  ICF/MR  beds. 

The  following  report  is  organized  into  nine  parts:  Part  II  includes  a  summary  of  the 
methodology  that  governed  the  various  project  activities;  Part  III  provides  a  summary 
of  the  findings  from  the  50  state  survey  of  quality  assurance  methods;  Part  IV  provides 
a  summary  of  the  findings  from  the  literature  review  of  outcome  and  other  research; 
Part  V  provides  a  summary  of  the  findings  and  themes  from  the  8  state  site  visits;  Part 
VI  provides  a  summary  of  the  results  of  the  reliability  and  validity  analyses  of 
outcome-based  state  and  accreditation  instruments  as  well  as  the  findings  from  the 
health  and  safety  surveys,  Part  VII  presents  findings  related  to  the  examination  of 
performance  indicator  data  bases;  Part  VIII  provides  an  analysis  of  the  application  of 
the  current  ICF/MR  active  treatment  regulations  and  their  relationship  to  outcome- 
focused  QA  approaches;  and  Part  XIX  includes  a  summary  of  conclusions  and 
recommendations. 
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II.  METHODOLOGY 


In  order  to  explore  the  ramifications  of  the  emerging  outcome  orientation  in  quality 
assurance  for  the  exercise  of  federal  regulatory  authority  under  the  regulations 
governing  Intermediate  Care  Facilities  for  People  with  Mental  Retardation  (ICFs/MR), 
the  Health  Care  Financing  Administration  contracted  with  Human  Services  Research 
Institute  (HSRI)  to  conduct  an  analysis  of  state  outcome  assessment  practices, 
performance  indicator  systems,  and  the  extent  to  which  current  the  active  treatment 
framework  in  federal  rules  is  consistent  with  burgeoning  outcome  oriented  certification 
approaches.  The  project  was  carried  out  in  collaboration  with  Temple  University 
Institute  on  Disabilities  and  Jaskulski  and  Associates.  To  complete  the  project,  three 
major  tasks  were  undertaken: 

•  Task  A:  Evaluation  of  the  ICF/MR  Program  Quality  Assurance  System; 

•  Task  B:   Identification  and  Evaluation  of  Quality  Indicators  (submitted  as  a 
separate  report); 

•  Task  C:  Establish  a  Baseline  of  Current  Psychoactive  Drug  Utilization  in 
ICFs/MR  (submitted  as  a  separate  report). 

To  ensure  advice  from  a  broad  array  of  constituencies,  a  Technical  Working  Group  was 
appointed  in  collaboration  with  the  HCFA  project  director.  The  Group  included 
representatives  of  major  advocacy  organizations,  accreditation  bodies,  provider 
agencies,  researchers,  and  family  members  and  people  with  mental  retardation. 

The  research  objectives  in  the  approved  research  plan  addressed  in  Task  A  are  as 
follows: 

•  To  summarize  the  "state  of  the  States"  in  outcome-based  quality  assurance  for 
ICF/MR  facilities  and  for  facilities /residences  serving  comparable  individuals; 

•  To  conduct  an  in-depth  review  and  analysis  of  such  systems  in  8  States; 

•  To  assess  the  reliability  and  validity  of  the  selected  quality  assurance  systems; 

•  To  identify  and  review  empirical  studies  of  outcome-focused  quality  assurance 
systems; 
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•  To  analyze  application  of  the  current  ICF/MR  active  treatment  definition  and 
its  consistency  with  current  concepts  of  support; 

•  To  identify  implications  for  changes  in  the  current  ICF/MR  quality  assurance 
system. 

The  methods  and  activities  carried  out  to  address  each  objective  are  described  below. 

Objective  1:  To  summarize  the  "state  of  the  States"  in  outcome-based  quality 
assurance  for  ICF/MR  facilities  and  for  facilities/residences  serving  comparable 
individuals. 

The  first  step  in  studying  the  "state  of  the  states"  was  to  collect  and  compile 
information  on  current  state  quality  assurance  approaches.  A  wide  range  of  published 
and  unpublished  sources  of  information,  covering  both  state  systems  and  other  quality 
assurance  systems  of  interest  was  canvassed.  Throughout  all  of  the  activities,  outcome- 
focused  systems  referred  to  systems  that  focus  on  the  primary  outcomes  of  interest  as 
outlined  in  HCFA's  Request  for  Proposal:  Individualization,  Integration/Inclusion, 
Relationships/Social  Connections,  Health  and  Safety,  Personal  Growth,  Self-determination, 
Dignity,  and  Consumer  Satisfaction 

A  telephone  inquiry  was  conducted  with  individuals  in  each  of  the  50  states  and  the 
District  of  Columbia.  Specifically,  the  survey  was  targeted  to  officials  at  the  state 
mental  retardation/ developmental  disabilities  department  most  knowledgeable  about 
the  state's  quality  assurance  system.  In  some  states  where  separate  QA  systems  were  in 
place  for  Medicaid  waiver  programs,  staff  also  contacted  individuals  in  the  state 
Medicaid  agency.  A  full  listing  of  all  key  informants  is  included  in  Appendix  A  of  this 
report.  Questions  from  an  inquiry  guide  were  asked  of  all  contacts  concerning  the 
extent  to  which  the  primary  outcomes  were  being  used,  what  data  are  collected  on 
primary  outcomes,  methods  and  measures  used  (including  information  about 
reliability  and  validity  of  the  measures)  and  effectiveness  of  outcome-focused  systems. 
The  results  of  these  inquiries  are  summarized  below  and  described  in  more  detail  in 
"Compendium  of  State  Outcome-Focused  Quality  Assurance  Systems,"  the  interim 
report  for  Task  A  submitted  to  HCFA  in  July,  1995. 

Objective  2:  To  conduct  an  in-depth  review  and  analysis  of  such  systems  in  6-8 
States. 

In  order  to  select  the  8  systems  for  in-depth  review,  the  following  criteria  were 
developed  in  conjunction  with  HCFA: 
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•  The  presence  of  a  quality  assurance  system  that  employs  measurements  of 
client  circumstances  that  coincide  with  the  selected  outcomes; 

•  The  availability  of  quality  assurance  information  regarding  health  and  safety 
as  well  as  client  outcomes; 

•  The  presence  of  a  data  collection  instrument  that  has  been  validated  and  tested 
for  reliability; 

•  The  presence  of  other  empirical  analyses  of  the  quality  assurance  data  base  that 
related  to  its  effectiveness  and/or  reliability; 

•  Application  of  a  quality  assurance  measure  on  clients  in  a  range  of  residential 
contexts  including  ICFs/MR  -  private  and  public; 

•  The  presence  of  an  innovative  quality  assurance  system  that  has  been  in  place 
for  at  least  a  year  and  preferably  longer; 

•  Willingness  of  state  and /or  agency  administrators  to  assist  in  validation  and 
reliability  process. 

•  The  presence  of  innovative  waiver  and  or  CSLA  QA  mechanisms. 


Though  no  state  met  all  of  the  criteria,  eight  states  had  a  sufficient  complement  of  the 
components  noted  above  and  were  proposed  as  potential  site  visit  states. 
Recommendations  were  made  to  members  of  the  Technical  Working  Group,  who 
ranked  their  selections  in  order  of  priority.  Initially,  six  states  were  chosen;  two 
additional  states  were  added  after  HCFA  provided  additional  resources.  The  eight 
states  visited  were  Massachusetts,  Missouri,  New  York,  Oklahoma,  Oregon,  Utah, 
Colorado  and  Wyoming.  The  State  of  Wyoming  was  subsequently  eliminated  from  the 
analysis  because  of  the  formative  nature  of  the  quality  assurance  system  and  the  lack  of 
any  systematic  data.  Prior  to  the  site  visits,  an  interview  guide  was  developed  which 
covered  the  following  areas: 

•  The  ways  in  which  the  quality  assurance  process  was  developed  and  the  extent 
to  which  it  represents  a  consensus  across  the  various  constituencies  in  the  state 
(e.g.,  providers,  clients,  families,  advocates,  etc.); 

•  Development  of  outcome-focused  quality  assurance  in  the  state,  including  the 
impetus,  level  of  consensus  and  service  system  trends 

•  The  outcomes  being  measured,  including  how  selected,  applicability  to 
individuals  with  severe  disabilities; 
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•  The  role  of  the  quality  outcome  system  in  the  states  residential  services  quality 
assurance,  including  how  related  to  compliance  standards  and  health  and 
safety  measures;  and  the  extent  to  which  current  QA  procedures  protect  the 
health  and  safety  of  persons  using  residential  services  and  supports; 

•  Perceived  effectiveness  of  the  state's  quality  assurance  approaches  and 
relevance  to  the  valued  outcomes; 

•  Use  of  quality  assurance  information  to  enhance  and  improve  services; 

•  Comparisons  of  experiences  with  the  ICF/MR  regulations  and  state  QA 
approaches. 

One  to  two  staff  from  HSRI  spent  two  to  three  days  in  each  state  conducting  interviews 
with  key  stakeholders  including:  state  officials,  state  quality  assurance  personnel, 
providers,  representatives  of  advocacy  organizations,  directors  and  staff  of  ICFs/MR, 
ICF/MR  surveyors,  and  individuals  who  live  in  residential  programs  and  their 
families.  Providers,  advocates  and  consumers  were  usually  interviewed  in  groups  and 
interviews  were  conducted  both  in  the  state  capitol  and  in  at  least  one  other  area  of  the 
state.  Site  visit  reports  were  drafted  and  sent  to  the  state  for  comments. 

In  addition  to  the  eight  states,  project  staff  also  reviewed  implementation  of  the  new 
(1993)  Outcome-based  Performance  Measures  developed  by  the  Accreditation  Council 
and  the  new  (1995)  standards  for  community  support  accreditation  from  CARF. 

In  addition  to  the  qualitative  assessments  in  each  state,  HCFA  has  required  that  "the 
contractor  shall  identify  the  degree  to  which  the  outcome  of  protection  of  beneficiary 
health  and  safety,  is  present  in  each  system."  In  order  to  respond  to  that  issue,  Temple 
developed  a  Health  and  Safety  Issues  instrument.  The  instrument  includes  information 
about  medications,  unusual  incidents,  behavior  programs  and  safety  within  the 
residential  setting.  Data  was  collected  in  each  of  the  eight  states  for  a  sample  of  at  least 
30  individuals.  Interviews  were  typically  conducted  with  direct  care  staff  who  knew 
the  individuals  well;  however,  whenever  possible,  individuals  were  invited  and 
encouraged  to  participate  in  the  interviews.  A  copy  of  the  Health  and  Safety 
Instrument  can  be  found  in  Appendix  B. 

In  addition  to  collecting  original  health  and  safety  data,  Temple  staff  reviews  collected 
other  data  with  which  the  health  and  safety  data  can  be  compared  including:  outcome 
data,  licensing  data,  OSCAR  data  (for  ICFs/MR)  and  any  other  quality  assurance  data 
collected  by  the  state  on  an  ongoing  basis.  The  purpose  for  gathering  these  data  is  to 
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determine  the  extent  to  which  states'  quality  assurance  systems  capture  information 
regarding  beneficiary  health  and  safety. 

Objective  3:  To  assess  the  reliability  and  validity  of  the  selected  quality  assurance 
systems. 

In  addition  to  the  qualitative  assessments  being  performed  as  part  of  the  Task  A 
analysis,  it  was  determined  by  both  the  contractor  and  HCFA  that  any  outcome-based 
system  that  is  recommended  contain  strong  statistical  properties  such  as  reliability  and 
validity.  In  particular  the  study  team  agreed  upon  the  following  statistical  measures  to 
assess  the  instruments  used  to  measure  quality  outcomes  for  individuals,  programs  and 
agencies. 

Reliability 

The  study  team  has  attempted  to  ascertain  the  extent  to  which  the  instruments  used 
possess  inter-rater  reliability;  the  extent  to  which  two  raters  score  information  about  an 
individual  in  the  same  way.  The  first  step  in  this  process  was  to  determine  whether 
states  had  completed  their  own  reliability  studies.  If  so,  those  studies  were  obtained 
and  reviewed.  In  those  cases  where  the  studies  established  statistically  significant 
reliability  coefficients,  the  reports  were  summarized  and  are  part  of  the  state 
summaries  of  quantitative  findings  in  Appendix  C.  In  those  states  where  reliability 
data  have  not  been  collected,  or  have  not  been  collected  in  several  years  (e.g.  Colorado) 
additional  data  collection  was  necessary.  To  complete  inter-rater  reliability  studies, 
typically  two  data  collectors  collect  information  about  the  same  individual  from  the 
same  respondent  with  a  relatively  short  period  of  time  elapsing  between  data 
collections.  In  order  to  complete  this  activity,  on-site  review  in  the  individual  state  was 
not  always  necessary.  In  some  cases,  the  instruments  used  in  each  of  the  states  could  be 
brought  to  Pennsylvania.  When  that  was  determined  to  be  appropriate,  the 
instruments  were  administered  in  the  residential  service  system  in  Pennsylvania  by 
two  separate  raters. 

From  a  research  perspective,  reliability  studies  are  also  helpful  in  that  they  often  point 
to  the  need  for  additional  or  more  consistent  training  of  interviewers,  to  insure  that 
data  are  collected  consistently,  regardless  of  who  collects  the  data.  Many  of  the  states 
and  entities  (AC  and  CARF)  have  developed  instrumentation  that  is  premised  upon 
extensive  training  and  re-training  of  data  collectors. 

As  we  have  determined  through  the  site  visits  to  the  eight  states  and  the  observation  of 
both  AC  and  CARF  surveys,  some  of  the  instruments  that  have  been  developed  require 
extensive  training  and  require  several  days  to  complete  the  ratings  for  one  individual. 
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Particularly  in  the  case  where  extensive  training  is  required,  (e.g.  AC,  CARF)  importing 
instruments  to  Pennsylvania  was  determined  to  be  inappropriate  (from  a  research 
perspective),  by  both  members  of  the  study  team  and  by  the.  individuals  responsible  for 
implementation  of  the  quality  assurance  systems  in  each  of  the  states.  Additionally,  the 
burden  of  asking  the  states/entities  to  perform  the  required  re-surveys  would  have 
been  excessive.  Fortunately,  in  many  cases  the  states/entities  have  already  embarked 
upon  reliability  studies  of  their  own.  The  data  from  those  studies  is  summarized  and 
included  in  the  state  summary  reports. 

As  part  of  the  outcomes  survey  process,  the  Accreditation  Council  pays  significant 
attention  to  the  issue  of  inter-rater  reliability.  As  part  of  each  outcome  survey,  data  are 
collected  by  at  least  two  reviewers  during  an  interview.  One  of  the  two  individuals 
leads  the  interview;  however,  both  individuals  complete  ratings  for  the  individual 
interviewed.  These  data  are  currently  being  collected  and  analyzed  by  Johns  Hopkins 
University.  Once  a  preliminary  report  is  issued,  we  will  review  it  and  share  it  with 
HCFA. 

CARF  has  not  yet  been  involved  in  the  process  of  establishing  reliability  of  the  outcome 
standards.  Because  of  the  intensity  of  the  accreditation  process  and  the  training 
assumptions  for  conducting  reviews,  the  Study  Team  determined  it  was  inappropriate 
to  conduct  a  reliability  study  of  CARF.  Similarly,  because  of  the  intensity  and  length  of 
the  process,  providers  were  unwilling  to  submit  to  additional  surveys  for  purposes  of 
this  study. 

For  the  states  where  Temple  has  conducted  the  reliability  study  (New  York  and 
Colorado),  data  were  collected  by  a  reviewer  trained  by  the  Study  Team. 
Approximately  two  to  four  weeks  after  the  initial  data  collection,  a  second  reviewer 
was  sent  to  complete  data  collection  instruments  with  the  same  respondent  used  for  the 
first  data  collection.  Data  were  collected  for  this  part  of  the  project  in  a  large,  public 
ICF/MR;  a  large,  private  ICF/MR;  and  several  small,  private  ICFs/MR. 

Concurrent  Validity 

In  this  portion  of  the  study,  the  study  team  is  measuring  the  extent  to  which  the 
instruments  used  measure  the  same  underlying  dimensions  as  other  instruments  that 
have  established  properties  of  reliability  and  validity.  The  instrument  chosen  for 
comparative  purposes  was  the  Behavior  Development  Survey,  an  instrument  used  and 
modified  by  Temple  University  for  more  than  15  years.  A  copy  is  included  in 
Appendix  B  of  this  report.  The  instrument  has  been  proven  reliable  (Devlin,  1981)  and 
includes  many  of  the  valued  outcomes  identified  by  the  HCFA  work  group  including: 
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•  Individualization; 

•  Integration/Inclusion; 

•  Relationships /Social  Connections; 

•  Health  and  Safety; 

•  Personal  Growth; 

•  Self-determination; 

•  Dignity;  and 

•  Consumer  Satisfaction. 

Again,  the  first  step  in  this  process  was  to  determine  whether  the  states  had  undertaken 
their  own  studies  of  concurrent  validity.  For  those  states  that  have  completed  studies, 
the  studies  have  been  requested,  received  and  evaluated  by  the  study  team.  If  they 
have  been  deemed  acceptable  (demonstrating  statistically  significant  correlations)  the 
studies  have  been  summarized  and  included  as  part  of  the  state  summaries  in 
Appendix  C.  For  those  states  where  concurrent  validity  has  not  been  tested,  data 
collection  has  occurred.  In  those  situations  where  Pennsylvania  staff  could  be  easily 
trained  to  apply  an  instrument  and  where  the  instrument  could  be  applied  in  a 
reasonable  amount  of  time,  the  instrument  was  imported  to  Pennsylvania  and 
administered  to  individuals  along  with  the  Temple  Behavior  Development  Survey 
(Temple  BDS).  The  two  instruments  would  then  be  correlated  with  one  another. 

Through  the  endorsement  of  this  project  by  the  Commonwealth  of  Pennsylvania's 
Office  of  Mental  Retardation,  the  Study  Team  received  access  to  individuals  living  in  a 
large,  publicly  operated  ICF/MR.  In  addition,  through  the  cooperation  of  a  private 
ICF/MR  provider  in  Pennsylvania,  the  study  team  was  given  access  to  individuals 
living  in  a  large,  private  ICF/MR  and  to  individuals  living  in  several  small  ICFs/MR. 
The  study  team  felt  it  was  important  to  collect  information  in  a  variety  of  settings.  The 
Temple  BDS  was  collected  along  with  the  Colorado  instrument  (COPAR)  and  the  New 
York  instrument  for  approximately  30  individuals;  10  in  each  type  of  setting. 

For  states  where  training  requirements  are  extensive  and  time  spent  is  long,  the  Temple 
(BDS)  was  exported  to  the  states,  where  trained  data  collectors  collected  Temple  BDS  in 
states  where  their  own  outcome  assessments  have  already  been  completed.  This 
occurred  in  both  Massachusetts  and  Missouri.  With  the  help  of  the  states  involved, 
HSRI  and  the  University  Affiliated  Program  in  Missouri,  Temple  identified  data 
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collectors  and  provided  training  to  them  by  phone.  For  AC  and  CARF  the  study  team 
pursued  the  possibility  of  collecting  Temple  BDS  data  in  agencies  that  have  received 
accreditation  with  the  organization's  outcome  measures.  Both  accrediting  bodies 
shared  with  the  Study  Team  a  listing  of  the  facilities  across  the  country  that  had 
received  accreditation  using  the  outcome  standards.  The  Study  Team  then  identified 
those  facilities  that  were  located  in  the  area  surrounding  Philadelphia,  or  located  in 
areas  where  there  was  a  pool  of  trained  data  collectors  (e.g.,  Missouri  and 
Massachusetts).  Agencies  were  contacted  and  from  the  contacts  made  we  were  able  to 
complete  data  collection  for  30  individuals  supported  by  an  AC-accredited  agency  and 
30  individuals  supported  by  a  CARF-accredited  facility.  Both  of  the  national  accrediting 
organizations  have  been  extremely  helpful  during  this  project,  both  in  facilitating  a 
member  of  the  Study  Team's  participation  in  an  accreditation  review  and  in  helping  the 
Study  Team  complete  the  concurrent  validity  portion  of  the  study. 

Temple  has  conducted  concurrent  validity  studies  for  Colorado,  Massachusetts, 
Missouri,  New  York,  AC  and  CARF.  Oregon  has  completed  its  own  concurrent 
validity  study  and  its  results  are  summarized  in  the  state  summary.  In  Utah,  an 
"informal"  study  was  conducted,  where  staff  from  Utah  went  to  Colorado  to  apply  the 
Outcomes  Instruments  to  sites  in  Colorado. 

In  Oklahoma,  a  concurrent  validity  study  was  deemed  unnecessary,  as  the  Study  Team 
was  involved  in  the  development  of  the  Oklahoma  instrument,  which  includes  most  of 
the  scales  that  are  being  utilized  in  the  Temple  BDS.  The  Temple  BDS  has  been  tested 
for  reliability  and  its  coefficients  have  been  found  to  be  within  acceptable  ranges. 
Therefore,  it  is  unnecessary  to  repeat  the  reliability  studies. 

Applicability 

The  purpose  of  this  portion  of  the  analysis  is  to  determine  the  extent  to  which  the 
instruments  can  be  used  in  a  variety  of  settings,  and  for  people  with  the  most  severe 
disabilities.  In  several  states  the  outcome  instruments  that  have  been  developed  are 
applied  across  all  residential  settings.  However,  in  some  states  the  instrument  is  only 
used  with  a  subgroup  of  the  population,  such  as  in  Colorado  where  the  COPAR  is  only 
used  in  agencies  that  contract  with  the  Community  Centered  Boards;  therefore 
ICFs/MR  are  not  included.  In  Utah  the  Outcome  Survey  is  only  used  in  HCB  Waiver 
programs  or  programs  that  are  100%  state  funded;  therefore  Utah  does  not  use  the 
instrument  in  ICFs/MR  or  in  settings  where  the  most  capable  people  live  (ICAP  scores 
of  level  1  or  2). 

For  those  instruments  that  are  only  used  with  a  subgroup  of  the  population,  the 
instruments  were  applied  in  a  large,  state-operated  and  a  large,  privately-operated 
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ICF/MR  in  Pennsylvania,  where  feasible.  Most  of  the  individuals  in  those  settings 
experience  severe  or  profound  mental  retardation,  many  use  wheelchairs  and  many  do 
not  communicate  in  traditional  ways.  In  those  states  where  applying  the  outcome 
instrument  requires  extensive  time  or  training,  the  instruments  were  not  brought  to 
Pennsylvania.  Wherever  possible,  anecdotal  data  were  collected,  indicating  whether 
the  instrument  is  being  used  with  individuals  with  the  most  significant  disabilities. 

In  Oklahoma  and  with  CARF,  the  outcome  instruments  are  used  in  large,  public 
ICFs/MR,  places  where  individuals  with  the  most  severe  disabilities  often  live.  In 
Massachusetts,  their  outcome  instrument  (the  QUEST)  has  been  pilot  tested  in  a  large, 
public  ICF/MR.  As  one  would  expect,  the  data  show  that  the  instrument  can  be  used, 
but  the  scores  in  areas  such  as  community  inclusion  are  significantly  lower  than  in  most 
community  settings.  Instruments  have  been  tested  by  the  Study  Team  for  applicability 
using  the  Colorado  instrument  (both  in  Pennsylvania  and  in  an  ICF/MR  in  Colorado) 
and  the  New  York  instrument. 

The  Missouri  instrument  is  used  only  in  the  HCB  waiver  program,  and  because  of  the 
length  of  time  it  takes  to  administer  and  the  data  collector  training  requirements,  was 
not  collected  by  the  Study  Team.  In  Utah,  the  outcomes  instrument  is  used  only  in 
waiver  or  wholly  state-funded  programs.  This  is  partially  due  to  the  fact  that  the  DD 
program  office  has  no  jurisdiction  over  the  ICF/MR  program.  In  addition,  because  of 
its  perceived  intrusiveness,  the  instrument  is  not  used  in  settings  that  support 
individuals  with  the  most  mild  disabilities  (defined  as  ICAP  level  1  or  2).  Because  of 
the  training  requirements  and  length  of  its  survey  process  AC  was  not  tested  for 
applicability.  Thus  far,  the  largest  facility  receiving  accreditation  utilizing  AC  is  a 
facility  supporting  close  to  200  individuals. 

The  Oregon  instrument  (Residential  Outcome  System  (ROS))  is  used  only  in 
community  settings.  However,  parts  of  the  ROS  have  been  used  by  one  of  the  two 
large  ICFs/MRs;  it  is  not  perceived  as  applicable  in  total  in  such  facilities.  However,  in 
collecting  health  and  safety  data  in  Oregon,  the  study  team  collected  data  in  a  home 
where  people  lived  who  have  significant  medical  complexities. 

Discriminant  Validity 

This  portion  of  the  study  determines  the  extent  to  which  the  instruments  are 
successfully  able  to  discriminate  between  individuals  such  that  their  scores  on  the 
outcomes  are  reflective  of  where  they  live,  for  example.  In  order  to  perform  this  part  of 
the  study,  no  original  data  collection  was  needed.  However,  the  study  team  did  need 
access  to  the  data  that  have  been  collected  on  the  quality  outcome  measures  in  each  of 
the  states.  For  the  most  part,  this  effort  went  smoothly;  however,  need  for  consent  to 
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release  information  was  a  lengthy  process  in  some  states.  In  a  few  states  the  data  that 
have  been  collected  have  been  collected  in  long  hand  only,  and  had  to  be  entered  onto 
the  computer  system  at  Temple. 

Data  were  received  and  analyzed  for  Colorado,  Massachusetts,  Missouri,  Oklahoma, 
Oregon  and  Utah.  In  Massachusetts  and  Missouri  data  are  collected  at  the  facility  level 
only;  therefore  analyses  will  be  performed  on  a  smaller  data  set  than  in  states  where 
data  are  maintained  for  individuals.  New  York  had  very  little  data,  so  the  data 
collected  for  the  reliability  and  concurrent  validity  portions  of  the  study  were  used  for 
the  discriminant  validity  portion  of  the  study.  Again,  detailed  results  are  included  in 
Appendix  C. 

Health  And  Safety 

In  addition  to  the  aforementioned  statistical  properties,  HCFA  has  required  that  "the 
contractor  shall  identify  the  degree  to  which  the  outcome  of  protection  of  beneficiary 
health  and  safety,  is  present  in  each  system."  In  order  to  respond  to  that  issue,  Temple 
developed  a  Health  and  Safety  Issues  instrument.  The  instrument  includes  information 
about  medications,  unusual  incidents,  behavior  programs  and  safety  within  the 
residential  setting.  The  instrument  was  collected  in  each  of  the  seven  states  for  a 
sample  of  at  least  30  individuals,  most  of  whom  had  outcome  data  collected  about 
them,  the  program  in  which  they  live,  or  the  agency  from  whom  they  receive  services. 
Interviews  were  typically  conducted  with  direct  care  staff  who  knew  the  individuals 
well;  however,  whenever  possible,  individuals  receiving  services  and  supports  were 
invited  and  encouraged  to  participate  in  the  interviews. 

In  order  to  compare  the  Health  and  Safety  data  collected  by  the  Study  Team  to  other 
Health  and  Safety  data  sources,  a  series  of  scales  were  created.  The  first  scale  is  called 
the  Medication  Scale.  One  point  was  given  for  each  of  the  following  situations: 

•  The  individual  receives  5  or  more  medications; 

•  The  individual  receives  medication  for  behavior  control; 

•  The  individual  receives  medication  for  behavior  control  but  a  continuing  need 
for  the  medication  has  not  been  certified  by  the  prescribing  physician; 

•  The  individual  receives  medication  for  behavior  control  but  there  has  been  no 
assessment  of  side  effects; 

•  The  individual  receives  medication  for  behavior  control  and  has  side  effects; 
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•  The  individual  receives  medication  for  behavior  control  and  has  no  psychiatric 
diagnosis; 

•  The  individual  receives  medication  for  behavior  control  and  has  no  behavior 
plan  that  includes  the  reduction  of  medication. 

The  Medication  Scale  has  a  range  of  0  to  7,  with  a  7  indicating  significant  problems  in 
that  area. 

The  next  area  examined  is  the  total  number  of  doctor's  visits  during  the  past  year. 
Respondents  were  asked  to  count  the  number  of  visits  to  general  practitioners, 
gynecologists,  psychiatrists,  dentists,  podiatrists  and  any  other  doctors  the  individual 
has  seen  during  the  past  year.  Respondents  were  asked  to  verify  visits  in  the 
individual's  record. 

The  next  scale  calculated  from  the  Temple  Health  and  Safety  instrument  is  called  the 
Behavior  Plan  Scale.  The  scale  is  comprised  of  the  following  items,  where  1  point  is 
given  where  each  situation  occurs: 

•  Evidence  of  physical  restraints; 

•  Evidence  of  restraints  that  are  not  part  of  a  program  plan; 

•  Evidence  of  a  time  out  room; 

•  Evidence  of  a  time  out  room  that  is  not  part  of  a  behavior  plan; 

•  Evidence  of  the  use  of  painful  or  noxious  stimuli; 

•  Evidence  of  the  use  of  painful  or  noxious  stimuli  that  is  not  part  of  a  program 
plan;  and, 

•  There  were  unusual  incidents  that  were  not  reported. 

The  scale  ranges  from  0  to  7,  with  a  0  indicating  no  issues  in  the  area  of  the  Behavior 
Plan  Scale  and  a  7  indicating  serious  problems  in  the  area. 

The  Safety  Scale  was  created,  using  the  following  10  items,  where  1  point  was  given 
for  each  occurrence: 

•  Inadequate  number  of  smoke  detectors; 
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•  Not  all  smoke  detectors  work; 


•  Not  all  fire  extinguishers  are  charged; 

•  There  was  no  fire  drill  in  the  previous  60  days; 

•  There  was  no  fire  drill  in  the  60  days  prior  to  that; 

•  The  home  or  living  area  is  not  clean; 

•  There  are  unpleasant  odors  in  the  home  or  living  area; 

•  There  are  physical  hazards  in  the  home  or  living  area; 

•  The  individual  does  not  have  all  prescribed  adaptive  equipment;  and 

•  Not  all  living/ program  areas  are  accessible. 

The  scale  ranges  from  0  to  10,  with  a  0  indicating  no  problems  in  the  area  of  safety  and 
a  10  indicating  significant  problems  in  that  area. 

Last,  the  Study  Team  examined  the  issue  of  neighborhood  safety.  One  item  was 
collected  assessing  neighborhood  safety.  The  item  asked,  how  safe  is  this 
neighborhood?  Possible  responses  were: 

0  Extremely  safe 

1  Safe 

2  Neither  safe  nor  unsafe 

3  Unsafe 

4  Extremely  unsafe 

Safety  is  a  quite  subjective  dimension.  Hence,  it  was  decided  to  leave  its  definition  and 
interpretation  to  the  respondent,  rather  than  to  the  data  collector  who,  not  having 
experience  in  the  specific  geographic  location,  may  define  safety  differently  from  the 
respondent  who  is  more  likely  to  know  the  neighborhood  well. 

In  addition  to  collecting  primary  source  health  and  safety  data,  Temple  staff  requested 
and  have  received  other  data  with  which  the  health  and  safety  data  have  been 
compared  including:  outcome  data,  licensing  data,  Onsite  Survey  and  Certification 
Automated  Reporting  (OSCAR)  data  (for  ICFs/MR)  and  any  other  quality  assurance 
data  collected  by  the  state  on  an  ongoing  basis.  The  purpose  for  gathering  these  data  is 
to  determine  the  extent  to  which  states'  quality  assurance  systems  capture  information 
regarding  beneficiary  health  and  safety. 
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For  those  facilities  that  carried  the  ICF/MR  designation,  results  of  the  ICF/MR  survey 
were  entered  onto  a  computer  system  that  can  then  be  accessed  by  states  and  by  the 
federal  regions.  The  data  are  presented  in  a  variety  of  standardized  formats  that 
include  data  such  as  deficiency  history,  which  lists  the  standards  for  which  the  facility 
was  deficient  over  a  period  of  several  years,  number  of  deficiencies  by  category,  and  so 
forth.  These  data  are  produced  in  reports  titled  Onsite  Survey  and  Certification 
Automated  Reports  (OSCAR).  For  the  states  where  ICFs/MR  were  reviewed  for 
purposes  of  the  health  and  safety  data  collection,  states  were  requested  to  provide 
copies  of  the  OSCAR3  and  OSCAR4  reports.  Where  states  were  unable  or  unfamiliar 
with  the  OSCAR  reports,  assistance  was  requested  and  obtained  from  the  HCFA 
Regional  Office. 

The  OSCAR  reports  that  were  requested  include  a  section  which  gives  an  analysis  of 
the  most  recent  ICF/MR  survey  by  deficiency.  The  deficiencies  are  grouped  by 
categories  that  include:  Conditions  of  Participation,  Standards  (which  includes  most  of 
the  programmatic  areas)  and  Life  Safety  Code.  For  each  of  the  categories,  the  actual 
number  of  deficiencies  is  recorded  for  the  facility.  A  table  is  included  to  show  the 
average  number  of  deficiencies  in  each  of  the  areas  for  the  state,  the  region,  and  the 
nation. 

After  consultation  with  HCFA  staff,  the  study  team  decided  to  use  the  nation  as  the 
area  for  comparative  purposes,  due  to  the  perceived  possibility  of  state  or  regional 
idiosyncrasies.  The  study  team  developed  a  scale  to  rate  each  of  the  categories  of 
deficiencies.  The  scale  is  as  follows: 

1  =  Much  Worse  -  the  facility  had  more  than  two  deficiencies  more  than  the 
national  average 

2  =  Somewhat  Worse  -  the  facility  had  more  deficiencies  than  the  national 
average 

3  =  Same  -  the  facility  had  the  same  number  of  deficiencies  as  the  national 
average 

4  =  Somewhat  Better  -  the  facility  had  fewer  deficiencies  than  the  national 
average 

5  =  Much  Better  -  the  facility  had  more  than  two  deficiencies  fewer  than  the 
national  average 

These  scales  (designated  01,  02,  and  03)  were  compared  with  the  states'  health  and 
safety,  quality  outcome  and  health  and  safety  data  that  were  collected  by  the  study 
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team.  For  Scale  01  (Conditions)  the  national  average  is  0.06  deficiencies.  For  Scale  02 
(Standards)  the  national  average  is  4.61  deficiencies.  Lastly,  for  Scale  03  (Life  Safety 
Code)  the  national  average  is  6.06. 

A  total  of  248  Health  and  Safety  Issues  surveys  were  completed  on  site  in  the  eight 
states  by  Temple  staff.  While  our  intent  was  to  do  30  Health  and  Safety  Surveys  in  each 
state,  ultimately  the  number  of  surveys  completed  varied  slightly  by  state.  The  number 
of  surveys  completed  by  state  were  as  follows: 


State 

#  interviews 

CO 

35 

MA 

30 

MO 

30 

NY 

30 

OK 

30 

OR 

31 

UT 

30 

WY 

32 

Total 

248 

The  types  of  residential  settings  visited  by  the  data  collectors  varied  by  state.  It  was  the 
Study  Team's  intent  to  include  a  sampling  of  as  many  of  the  different  residential 
options  as  possible.  However,  given  the  limited  amount  of  time  spent  in  each  state  by 
the  Study  Team,  not  all  residential  placement  types  were  represented.  No  small 
ICFs/MR  were  included  in  the  data  collection  for  a  variety  of  reasons.  In  some  states 
(Massachusetts  and  Missouri)  the  quality  assurance  systems  are  not  implemented  as 
part  of  the  ICF/MR  program.  In  other  states  (Colorado,  New  York  and  Oklahoma) 
most  of  the  small  ICF/MR  programs  are  being  converted  to  the  HCBW;  therefore,  there 
were  too  few  people  from  whom  to  choose.  Neither  Utah  nor  Oregon  participates  in 
the  small  ICF/MR  program  in  the  community.  ICF/MR  (16+  beds)  were  included  in 
Colorado,  Massachusetts,  Oklahoma  and  Utah.  The  breakdown  of  residential  types 
visited  (by  state)  is  as  follows: 
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Table  1-1 
Residential  Settings  by  State 


STATE 

ICF/MR 
16+ 

Group 
Home 

Supported 
Living 

Foster 
Family 

Family/ 

Own 

Home 

CO 

40% 

57% 

3% 

- 

- 

MA 

30% 

70% 

- 

- 

- 

MO 

- 

97% 

- 

- 

3% 

NY 

- 

100% 

- 

- 

- 

OK 

60% 

40% 

OR 

100% 

UT 

33% 

67% 
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The  level  of  mental  retardation  was  gleaned  from  the  records  as  part  of  the  data 
collection.  The  breakdown  by  state  is  as  follows: 


Table  1-2 
Level  of  Mental  Retardation  by  State 


STATE 

Mild 

Moderate 

Severe 

Profound 

CO 

14% 

20% 

34% 

32% 

MA 

30% 

20% 

30% 

20% 

MO 

36% 

23% 

37% 

3% 

NY 

17% 

13% 

23% 

47% 

OK 

40% 

13% 

20% 

27% 

OR 

33% 

27% 

10% 

30% 

UT 

30% 

20% 

37% 

13% 

Overall/ 
ICF/MR 

15% 

15% 

22% 

48% 

As  the  table  shows,  in  some  states  the  sample  included  more  people  with  mild 
retardation  (Oklahoma  and  Missouri).  In  other  states,  the  sample  included  more 
people  with  profound  retardation  (New  York). 

One  of  the  questions  asked  regarding  medical  status  was,  "In  general,  how  urgent  is 
this  person's  need  for  medical  care?"  The  possible  responses  were:  Would  not  survive 
without  24-hour  medical  care;  Has  life- threatening  condition  that  requires  very  rapid 
access  to  medical  care;  Needs  visiting  nurse  and/or  regular  visits  to  the  doctor;  and 
Generally  has  no  serious  medical  needs.  The  responses  by  state  were  as  follows: 
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Table  1-3 
Medical  Needs  by  State 


STATE 

Needs 
24  -hour 
care 

Needs 
Rapid 
Access 

Regular 
Visits  to 
Doctor 

No 

Serious 
Needs 

CO 

0% 

6% 

26% 

68% 

MA 

0% 

3% 

37% 

60% 

MO 

0% 

0% 

20% 

80% 

NY 

0% 

3% 

23% 

73% 

OK 

0% 

13% 

3% 

83% 

OR 

10% 

0% 

51% 

39% 

UT 

0% 

7% 

37% 

56% 

As  the  table  demonstrates,  most  of  the  individuals  in  the  survey  have  no  serious 
medical  needs.  Interestingly,  only  individuals  in  Oregon  were  reported  as  needing  24- 
hour  medical  care.  Those  individuals  are  currently  living  in  community  settings.  No 
one  that  was  surveyed  who  is  living  in  a  large  facility  was  judged  to  need  24-hour 
medical  care. 

Objective  4:  To  identify  and  review  empirical  studies  of  outcome-focused  quality 
assurance  systems, 

A  literature  search  was  conducted  using  four  data  bases:  PsychlNFO,  Educational 
Resources  Information  Center  (ERIC),  NTIS  and  Health  Planning  and  Administration. 
Relevant  studies  were  obtained  and  reviewed.  In  addition  to  published  research  on 
outcome  studies,  unpublished  studies  were  obtained. 
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The  project  team  undertook  two  activities  to  identify  the  efforts  nationwide  examining 
"quality  of  life"  outcomes  for  individuals  with  developmental  disabilities  along  some 
or  all  of  the  following  domains:  self-determination,  dignity,  health  and  safety, 
consumer  satisfaction,  individualism,  personal  growth,  community 
integration/inclusion,  and/or  social  relationships.  These  activities  were: 

1.  A  comprehensive  literature  search  using  CDP  On-Line  which  includes  more 
than  120  databases  including  PsycINFO,  Educational  Resources  Information 
Center,  National  Technical  Information  Center,  and  Health  Planning  and 
Administration; 

2.  The  collection  of  unpublished  empirical  studies  of  quality  outcomes,  conducted 
by  individual  states  and/ or  research  organizations. 

Presented  here  are  the  project  team's  findings  from  these  efforts,  followed  by  a 
summary  and  examination  of  the  findings,  and  their  implications  for  future  quality  of 
life  evaluations. 

Literature  Search 

A  comprehensive  on-line  literature  search  was  completed  to  identify  empirical  studies 
of  quality  of  life  outcomes  completed  within  the  past  ten  years.  The  following  four 
databases  were  searched: 

1.  PsycINFO  (PSYC),  which  covers  the  professional  and  academic  worldwide 
literature  of  psychology  and  related  fields,  includes  1300  journals,  book 
chapters  and  books,  contains  about  one  million  citations,  dating  back  to  1967; 

2.  Educational  Resources  Information  Center  (ERIC),  the  largest  educational 
database  in  the  world,  contains  over  800,000  citations,  dating  back  to  1966; 

3.  National  Technical  Information  Database  (NTIS),  the  major  resource  for 
locating  U>S>;  Government  sponsored  research  reports  in  physical,  health, 
social,  and  life  sciences,  contains  1;6  million  documents,  from  1970  to  the 
present; 

4.  Health  Planning  &  Administration  (HLTH),  an  information  resource  for 
medical  and  health  care  professionals,  covers  worldwide  literature  concerning 
health  care  planning,  organization,  financing,  management,  manpower,  and 
related  subjects,  from  1975  to  present; 
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Terms  used  during  the  search  included:  disabilities  (mental  retardation,  developmental 
disabilities),  studies  (empirical,  quantitative),  and  quality  (outcomes,  quality  of  life, 
quality  assurance,  quality  enhancement,  independent  living,  deinstitutionalization).  In 
this  effort,  however,  the  project  team  found  very  little  "new"  information.  In  fact,  most 
of  the  Quality  Assurance  literature  was  written  by  people  who  are  either  project  staff  or 
members  of  the  Technical  Working  Group.  No  "new"  Quality  Assurance  systems  (i.e. 
ones  we  did  not  already  know  about)  were  identified  in  the  literature  search. 

Published  And  Unpublished  Empirical  Studies  Of  Quality  Outcomes 

Through  discussions  with  state  administrators,  researchers,  and  members  of  the 
project's  Technical  Working  Group,  several  states  were  identified  as  participating,  to 
some  extent,  in  evaluation  efforts  using  quality  of  life  outcomes  as  a  standard  of 
measure.  To  some  extent,  these  discussions  assisted  in  the  determination  of  which 
states  participated  in  the  project's  site  visit  activities,  and  many  of  the  materials 
collected  were  gathered  during  site  visits  to  these  states. 

Data,  reports,  and  sometimes,  analyses  of  results  from  these  evaluation  efforts  were 
collected  from  the  states  of  Wyoming,  Oklahoma,  Oregon,  Utah,  Pennsylvania,  and 
Colorado.  Additionally,  a  recent  study  completed  by  The  Accreditation  Council  for 
HCFA,  analyzing  national  data  collected  during  the  ACID'S  Quality  Review  and 
accreditation  process,  was  obtained.  An  earlier  anticipated  evaluation  of  the  1115 
demonstration  in  Minnesota,  using  outcome-focused  regulations  and  monitoring  in 
ICFs/MR  and  Waiver  homes,  has  been  delayed  and  results  are  not  yet  available. 

On  the  topic  of  quality  of  life,  the  bulk  of  existing  literature  revolves  around:  1)  why  it 
is  important  to  look  at  quality  of  life  issues;  2)  redirecting  current  quality  assurance 
efforts  to  focus  on  quality  of  life  outcomes;  3)  defining  "quality  of  life";  and  4) 
developing  and  analyzing  methods  of  measurement.  Indeed,  very  little  has  been 
written  summarizing  or  analyzing  empirical  quality  outcome  studies  for  people  with 
developmental  disabilities.  Most  information  in  this  area  exists  only  in  unpublished 
articles,  and  is  available  primarily  through  state  disability  agencies. 

What  follows  below  is  an  overview  of  the  materials  identified  through  this  project's 
activities,  and  their  efforts  to  measure,  report,  and  analyze  results  of  local  quality  of  life 
outcome  studies. 

•  Pennsylvania  -  The  Pennhurst  Longitudinal  Study  (Conroy  &  Bradley,  1985),  a 
five-year  analysis  of  the  impact  of  relocating  people  with  developmental 
disabilities  into  the  community,  is  considered  the  bellwether  of  quality  outcome 
studies  and  has  been  used  as  a  foundation  by  other  states  redesigning  their 
quality  assurance  efforts.     The  primary  areas  of  consideration  for  this 
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evaluation  were  skill  &  behavioral  development,  consumer  satisfaction, 
qualities  of  environments  (physical  comfort,  18  aspects  of  normalization, 
individualization,  life  safety,  encouragement  of  autonomy  and  activity,  size 
and  staffing  patterns),  family  impacts,  and  costs.  Comparisons  were  made 
between  those  individuals  remaining  in  Pennhurst  and  those  relocating  to  the 
community.  Since  the  original  study,  data  continues  to  be  collected  and 
reported  annually  for  individual  counties. 

Oklahoma  —  The  Longitudinal  Assessment  of  Consumer  Outcomes  (Murray, 
Conroy  &  Ervin,  1994)  summarizes  an  evaluation  of  consumer  outcomes  in  the 
areas  of  independence  (adaptive  behavior),  productivity  (participation  in 
activities  that  are  useful  and  fulfilling),  integration  (opportunities  to  socialize 
with  general  population),  and  satisfaction.  Comparisons  by  characteristics  of 
consumer  and  residential  setting  (public  ICFs/MR,  private  ICFs/MR,  nursing 
facilities,  group  homes,  supported  living,  and  community  placements)  were 
conducted.  The  study  was  found  to  yield  generally  reliable  results,  except  for 
the  physical  quality  scale  and  site  impression  scale  (which  were  considered 
highly  subjective). 

Colorado  —  A  Report  on  Outcomes  of  Services  and  Supports  for  Persons  with 
Developmental  Disabilities  (Ruth  &  Struxness,  1994)  summarizes  and  analyzes  the 
results  of  ongoing  use  of  the  COPAR  (Colorado  Progress  Assessment  Review) 
instrument.  This  evaluative  tool  focuses  on  consumer  satisfaction  with  services, 
decision-making,  community  inclusion,  community  integrated  employment, 
talents  and  personal  goals,  relationships  and  a  sense  of  belonging,  personal 
security,  self-respect.  These  issues  directly  correspond  to  legislative  language 
pertaining  to  outcomes  for  people  with  developmental  disabilities.  In  1992-93, 
the  COPAR  was  administered  to  781  adults  with  disabilities,  and  1000  adults  in 
the  general  Colorado  population.  Comparisons  of  results  were  conducted 
along  four  basic  themes:  1)  comparison  of  outcomes  across  various  service 
approaches  (institutional  setting,  community  group  settings,  community 
individualized  settings);  2)  comparison  of  actual  service  performance  (all 
residential  service  participants)  against  a  performance  target  (general 
population);  3)  comparison  of  outcomes  longitudinally  to  see  if  there  are 
system-wide  improvements  in  outcomes  across  time;  and  4)  comparison  of 
outcomes  by  characteristics  of  people  in  services  (demographics,  support 
needs).  Tests  for  reliability /statistical  significance  were  conducted. 

Utah  -  Utah  annually  collects  and  reports  data  from  their  statewide  quality 
assurance  efforts.  The  1995  Consumer  Report  on  Home  and  Community  Based 
Services  (Division  of  Services  for  People  with  Disabilities,  1995)  summarizes 
outcome  surveys  yielding  scores  on  four  standards:  1)  opportunities  for 
personal  growth  and  development;  2)  facilitation  of  individualization  and  full 
participation;  3)  community  integration  (opportunities  to  live,  work  and 
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associate  with  others  in  the  community);  and  4)  use  of  a  provider  quality 
assurance  system.  The  report  also  includes  results  of  consumer  satisfaction 
surveys  (satisfaction  with  service  provider)  in  the  areas  of  day  training, 
supported  employment,  residential  services,  supported  living,  family  support 
and  the  professional  parent  program;  as  well  as  results  from  volunteer 
monitoring  visits. 

Oregon  -  Focusing  on  Values  and  Lifestyle  Outcomes  in  an  Effort  to  Improve  the 
Quality  of  Residential  Services  in  Oregon  (Newton,  Ard,  Horner  &  Toews  1996) 
details  efforts  to  assess  the  physical  integration,  independence,  and  social 
integration  of  individuals  with  developmental  disabilities  residing  in  Oregon's 
community  residential  settings.  Also  addressed  are  individual's  frequency  and 
variety  of  activities,  and  participation  in  preferred  activities.  Reporting  of 
results  centers  on  these  latter  activity  levels. 

Wyoming  —  Evaluation  efforts  in  Wyoming  are  illustrated  in  Changing  the  Shape 
of  the  Service  Population  (Fortune,  Heinlein  &  Fortune,  1995),  where  the  ICAP 
(Inventory  for  Client  and  Agency  Planning)  was  used  to  evaluate  adaptive 
behavior  (social  and  communication,  personal  living,  community  living,  and 
motor)  and  maladaptive  behavior  (internalized,  externalized,  and  asocial)  of 
people  with  developmental  disabilities  residing  in  the  state  institution  and 
community  settings.  The  study  period  included  years  1988  through  1994,  and 
compared  individuals  residing  in  institutions,  with  those  living  in  the 
community  and  those  transitioning  from  institutional  to  community  life.  Over 
the  years  noted,  this  study  examined  changes  in  the  distribution  of  ICAP  scores 
in  the  population  served,  and  changes  in  individual's  adaptive  and 
maladaptive  behavior  scores. 

The  Accreditation  Council  —  The  Quality  Assurance  Measures  Database:  Final 
Report  (ACD,  1995)  provides  an  overview  of  results  and  analysis  from 
interviews  conducted  with  440  individuals  with  disabilities  receiving  services 
from  54  organizations  across  the  country.  These  interviews  focused  on  the 
progress  toward  or  achievement  of  30  specific  "outcomes  for  people",  that  fell 
into  these  categories:  personal  goals,  choice,  social  inclusion,  relationships, 
rights,  dignity  and  respect,  health,  environment,  security,  and  satisfaction. 
Comparisons  of  results  were  examined  by  type  of  residential  setting,  funding 
(e.g.,  ICF/MR,  HCBW),  level  of  disability  (e.g.,  mild,  moderate, 
severe /profound  mental  retardation),  services  and  support  received, 
organization  size,  and  geographic  characteristics  (e.g.,  urban,  rural,  suburban). 

Pennsylvania  -  The  Small  ICF/MR  Program:  Dimensions  of  Quality  and  Cost 
(Conroy,  1996),  compares  quality  of  life  dimensions  for  51  people  living  in 
small  ICFs/MR  with  those  for  an  equal  number  of  individuals  living  in 
community  living  arrangements  (CLAs).  Data  for  this  analysis  was  collected 
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between  1988  and  1992,  using  49  indicators  of  quality  for  comparison  (e.g., 
choicemaking,  productivity,  home  environment,  health  and  health  care, 
integration,  behavioral  progress). 

Objective  5:  To  analyze  application  of  the  current  ICF/MR  active  treatment 
definition  and  its  consistency  with  current  concepts  of  support 

In  order  to  make  determinations  about  those  aspects  of  HCFA's  regulatory  definition  of 
active  treatment  and  interpretive  guidelines,  HSRI  used  several  sources  of  information 
in  conducting  the  review: 

•  Informal  discussions  with  State  contacts  across  the  nation  regarding  their 
experience  with  ICF/MR  quality  assurance  and  perspectives  on  the  active 
treatment  regulations  and  guidelines,  in  conjunction  with  the  inquiries  done  for 
the  development  of  the  Task  A  Interim  Report; 

•  Interviews  with  knowledgeable  individuals  during  the  State  site  visits 
conducted  to  review  outcome-focused  quality  assurance  systems; 

•  Review  of  the  literature  on  ICF/MR  quality  assurance; 

•  Input  from  members  of  the  Technical  Working  Group  and  their  respective 
organizations,  as  appropriate; 

•  Interviews  with  ICF/MR  providers  around  the  country; 

•  Review  of  HCFA's  Draft  Revisions  to  the  Survey  Process  for  ICFs/MR. 

In  consultation  with  the  HCFA  project  director,  project  staff  substituted  a  case  study 
approach  for  the  observation  of  an  ICF/MR  survey.  The  case  studies  of  active 
treatment  are  targeted  to  five  issues:  promotion  of  best  practice,  continuous  active 
treatment,  overprogramming,  choice,  and  conflicts  with  outcome-based  approaches. 

Objective  6:  To  identify  implications  for  changes  in  the  current  ICF/MR  quality 
assurance  system. 

The  results  of  all  of  the  above  activities  conducted  as  part  of  Task  A  as  well  as  feedback 
from  the  Technical  Working  Group  and  HCFA  were  reviewed  in  order  to  identify 
implications  for  changes  in  the  current  ICF/MR  quality  assurance  system. 
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HI.  RESULTS  AND  FINDINGS 


Objective  1:  Summary  of  Outcome  Focused-Quality  Assurance  Approaches 
The  Degree  Of  Quality  Assurance  Reform 

Quality  assurance  systems  are  in  a  period  of  transition  in  most  of  the  states,  each  at 
various  stages  of  reform.  Overall,  there  is  a  clear  trend  toward  outcome-focused 
quality  assurance  systems. 

Three  categories  were  developed  to  characterize  the  degree  to  which  quality  assurance 
reform  approaches  an  outcome-focused  system,  as  well  as  two  "transition"  categories. 
The  need  for  the  "transition"  categories  became  apparent  in  the  discussions  with  the 
States,  reflecting  both  the  strong  interest  in  quality  assurance  reform  and  the  current 
fluidity  of  quality  assurance  nationally.  A  total  of  29  States  also  reported  that  they  were 
projecting  changes  in  the  next  year  (i.e.,  1995-1996).  As  noted  above,  all  but  12  States 
are  actively  moving  toward  an  outcome-focused  quality  assurance  system  and  State 
officials  in  six  of  these  States  expressed  interest  in  increasing  their  focus  on  outcomes. 

The  categories  and  number  of  States  in  each  category  as  of  spring  1995  are  as  follows: 

Category  I:  Significant  movement  toward  outcome-focused  quality  assurance:  (10 
States) 

This  category  includes  States  that  may  not  have  completely  eliminated  traditional 
quality  assurance,  comprehensive  licensure  and  certification  (L&C)  surveys,  etc.,  but 
include  definitive  outcome  focus  and  systems  include  the  primary  outcomes  of  interest. 
These  States  may  not  have  included  all  settings  (e.g.  their  ICFs/MR  and  state 
institutions),  but  are  using  outcome-focused  quality  assurance  in  settings  serving 
individuals  with  the  same  characteristics  (e.g.,  severe  and  profound  mental  retardation, 
complex  disabilities). 

Transition  II  to  I:  States  actively  in  transition  to  an  outcome-focused  system:  (12 
States) 

This  category  includes  States  that  have  made  the  decision  to  reform  their  quality 
assurance  systems  to  an  outcome-focused  system,  and  the  reform  is  underway.  For 
example,  some  States  in  this  category  are  conducting  a  demonstration  to  determine 
how  to  implement  such  a  system. 
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Category  II:  Some  reform  aimed  at  outcome-focused  quality  assurance  including  the 
primary  outcomes  of  interest:  (9  States) 

States  in  this  category  may  be  applying  the  primary  outcomes  in  some  form  of  a 
demonstration  project,  such  as  a  pilot  in  one  part  of  the  State  that  may  become  the 
future  model,  or  an  outcome-focused  quality  enhancement  program  such  as  citizen 
monitoring,  or  an  outcome-focused  quality  assurance  that  applies  only  to  one  type  of 
residential  services  (for  example,  supported  living  -  if  possible,  however,  one  that 
includes  at  least  some  individuals  with  same  characteristics  as  ICF/MR  residents.) 
These  States  continue  to  rely  on  traditional  licensure  and  certification  process. 

Transition  III  to  II:  Beginning  the  process  of  developing  some  aspects  of  an 
outcome-focused  quality  assurance  system:  (5  States) 

States  in  this  category  are  actively  planning  a  pilot  project,  may  have  a  task  force  that 
has  recently  issued  its  first  report,  new  legislation  that  will  require  them  to  move  ahead 
with  reform,  but  are  not  committed,  as  yet,  to  comprehensive  quality  assurance  reform 
focused  on  outcomes. 

Category  III:  Little  or  no  movement  toward  outcome-focused  quality  assurance 
reforms:  (12  States) 

These  States  may  have  a  task  force  beginning  to  address  the  issues,  a  pilot  that  is  no 
longer  operative,  some  token  activities  focused  on  quality  of  life  measures,  but  are  still 
relying  primarily  on  traditional  quality  assurance. 

Other.  (3  States) 

In  this  category  are  States  with  deemed  status  provisions  (CARF  and /or  Accreditation 
Council)  [2  States]  and  one  that  is  moving  from  Category  I  to  Category  II. 

Table  IV  1  illustrates  the  category  of  quality  assurance  reform  for  each  State. 


28 


TABLE  2-4 

CATEGORY  OF  QUALITY  ASSURANCE  REFORM 


Stale 

I 

n>»i 

n 

m>»n 

m 

Other 

AL 

X 

AK 

X 

AZ 

X 

AR 

X 

CA 

X 

CO 

X 

CT 

X 

DE 

X 

FL 

X 

GA 

X 

HI 

X 

ID 

X 

DL 

X 

IN 

X 

IA 

X 

KS 

X 

KY 

X 

LA 

X 

ME 

X 

MD 

X 

MA 

X 

MI 

X 

MN 

X 

MS 

X 

MO 

X 
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TABLE  1-4  (CONTINUED) 


Stale 

I  ' 

n>»i 

n 

m»>n 

m 

Other 

MT 

X 

NE 

X 

NV 

X 

NH 

X 

NJ 

X 

NM 

X 

NY 

X 

NC 

X 

ND 

X 

OH 

X 

OK 

X 

OR 

X 

PA 

X 

RI 

X 

SC 

X 

SD 

X 

TN 

X 

TX 

X 

UT 

X 

VT 

X 

VA 

X 

WA 

X 

WV 

X 

WI 

X 

WY 

X 

DC 

X 

TOTAL 

10 

12 

9 

5 

12 

3 
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Scope  Of  Facilities/Residential  Services  And  Populations  Covered 

At  least  19  States  with  outcome-focused  quality  assurance  systems  apply  measures  to 
agencies  serving  significant  numbers  of  individuals  with  the  same  characteristics  as 
ICF/MR  residents  (severe  or  profound  levels  of  mental  retardation  and  multiple 
disabilities). 

All  but  five  States  that  are  using  outcome-focused  quality  assurance  approaches  in  a 
systematic  fashion  exclude  ICFs/MR  from  the  process  primarily  because  of  the 
traditional  separation  between  ICF/MR  quality  assurance  and  non-ICF/MR  quality 
assurance.  As  illustrated  in  IV  2,  however,  some  States  do  include  ICFs/MR  in  their 
quality  assurance  activities: 


31 


TABLE  1-6 

SCOPE  OF  RESIDENTIAL  SERVICES  AND  POPULATIONS  COVERED 


State 

Scope  of  Residential 
Services  Covered 

Coverage  of  Individuals  with 
Same  Characteristics  as 
ICF/MR  Residents 

Additional 
Comments 

AK 

All  but  ICFs/MR  (one 
small  State  facility  and  five 
small  private  ICFs/MR) 

Yes 

CO 

Includes  State  institutions 
(ICFs/MR)  but  not 
community  ICFs/MR 
except  indirectly;  adults 
only 

Yes 

Because  COPAR  survey 
includes  sample  of 
community  day  program 
participants,  some 
community  ICF/MR 
residents  included  as  well 
as  State  institution 
residents 

FL 

Pilot  demo  in  one  State 
HRS  District;  includes 
HCB  waiver  but  no 
ICFs/MR 

Similar  functioning  level, 
however,  ICFs/MR  serve  those 
who  are  medically  fragile 

ME 

All  community  residences 

Yes 

MA 

All  community  residences, 
including  many  HCB 
waiver  (State  has  no 
community  ICFs/MR); 
using  in  State  institutions 
for  quality  enhancement 
and  piloted  QUEST  in 
State  institution 

Yes 

MN 

Demonstration  project 
with  5  ICF/MR  providers 
in  5  counties 

Yes,  although  community 
ICFs/MR  may  serve 
somewhat  less  severe/complex 
disabilities  than  State 
institutions 

Participating  agencies  will 
apply  the  new  quality 
assurance  to  150  ICF/MR 
residents  +  an  additional 
300  in  HCB  or  Supported 
Independent  Living 
services 
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TABLE  1-6  CONTINUED 


MO 

Primary  focus  on  HCB 
waiver  services 

Yes 

NH 

All  community  residences 
(NH  has  no  State 
institution) 

Yes 

Mixed 

NY 

COMPASS  demonstration 
project  with  24  OMRDD- 
licensed  community 
residential  service 
agencies  /l  ,000  individuals, 
including  both  State-  and 
privately-operated.  A  few 
ICF/MR  and  HCB  waiver 
beneficiaries  included 
indirectly. 

Yes  -  24  percent  with  severe 
MR,  7  percent  with  profound 
MR;  also  14  percent  with 
physical  disabilities  and /or 
"wheelchair  dependent" 

State  plans  to  apply  for 
demonstration  of 
COMPASS  in  community 
ICFs/MR  and  to  extend  to 
HCB  waiver  services  in  the 
near  future 

OH 

All,  including  State 
institutions 

Yes 

OK 

All  community  residential 
services;  longitudinal 
outcome  data  also  being 
collected  in  State 
institutions 

Yes 

OR 

All  except  State 
institutions,  however,  they 
use  for  QE 

Yes 

OR- AIM  (citizen 
monitoring  focused  on 
outcomes,  quality  of  life) 
required  for  people 
moving  from  State 
institution  to  community 

RI 

All,  including  State 
institutions 

Yes 

TN 

All  except  ICFs/MR 

Yes 

UT 

All  except  ICFs/MR 

Yes 

VT 

All  community  residences 

Yes 
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TABLE  1-6  CONTINUED 


WI 

HCB  waiver  services 

Yes 

Targeted  especially  to 
individuals  who  have 
moved  from  State 
institution  to  community 
residences 

WA 

All  except  ICFs/MR 

Yes  -  500  of  the  900  people 
who  have  moved  from  the 
State  institution  and  whose 
outcomes  are  monitored  are 
individuals  with  either  severe 
or  profound  mental 
retardation. 

WY 

All  except  the  State 
institution  (no  other 
ICFs/MR) 

Yes 
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Selection  Of  Quality  Outcomes 


The  project  survey  shows  that  States  that  have  moved  toward  an  outcome-focused 
quality  assurance  system  and  have  defined  important  outcomes  as  part  of  that  effort 
have  selected  outcomes  that  are  remarkably  consistent  with  those  selected  by  HCFA 
and  the  Technical  Working  Group  (the  primary  outcomes).  The  States  with  all  eight 
primary  outcomes  are  illustrated  in  Table  IV  3. 

Table  1-7 


Selection  of  Primary  Outcomes 


Primary  Outcome 

Number  of 
States 

How  primary  outcome  Typically  Defined 

Individualization 

30 

Services  based  on  individual  choices, 
preferences,  needs  -  primary  emphasis  on 
choice  (22  of  the  30) 

Integration/Inclusion 

29 

Community  presence,  participation, 
membership;  social  inclusion 

Relationships/Social 
Connections 

26 

Relationships  with  friends  and  family 

Health  and  Safety 

25 

Health  and  safety;  well-being;  personal 
security 

Personal  Growth 

25 

Various  -  personal  growth,  competence, 
attainment  of  individual  goals,  skill 
acquisition 

Self-determination 

25 

Various  -  control,  choice-making,  decision- 
making, power,  independence 

Dignity 

24 

People  treated  with  respect  and  dignity; 
age-appropriate  services  (some) 

Consumer  Satisfaction 

19 

Consumer  satisfaction 
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Other  observations  include: 

•  The  majority  of  state  quality  assurance  systems  do  not  include  a  database  on 
the  primary  outcomes  of  quality; 

•  Decreasing  focus  on  document  review  with  a  corresponding  increasing  focus 
on  observation  and  interviews,  in  particular  increased  contact  with  service 
recipients; 

•  Continuing  use  of  process  measures  in  outcome-focused  quality  assurance 
reviews,  but  with  greater  emphasis  on  processes  that  are  seen  as  directly  tied  to 
specific  primary  outcomes; 

•  A  mixed  picture  as  to  how  much  "traditional"  quality  assurance  is  being 
retained  as  primary  outcomes,  in  particular  those  associated  with  quality  of  life; 

•  General  trend  toward  reduction  of  prescriptive  rules  and  standards  and 
development  of  streamlined  licensure  and  certification  procedures; 

•  Increased  focus  on  individuals,  their  satisfaction,  and  choices,  as  well  as  an 
emerging  approach  with  individuals  as  the  organizing  principle  of  the  quality 
assurance  system,  (and  the  service  system)  rather  than  provider  agencies; 

•  Consensus  on  quality  assurance  reform  growing  among  stakeholders  in  most 
States,  and  recognition  of  the  importance  of  involving  stakeholders  in 
developing  and  building  consensus  on  outcome-focused  quality  assurance 
systems; 

•  Emerging  trends:  increased  decentralization  of  quality  assurance  activities, 
tying  quality  assurance  reforms  to  managed  care  and  using  cross-disability 
quality  indicators. 

Overall,  there  is  strong  interest  and  support  for  outcome-focused  quality  assurance 
systems  in  residential  services,  for  individuals  at  all  levels  of  mental 
retardation/developmental  disability,  and  in  all  settings.  The  above  findings  were 
mirrored  in  the  eight  case  study  states. 
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Objective  2:  State  Approaches  to  Outcome-Focused  Quality 
Assurance  in  Selected  States 

Overview 

The  seven  study  states  employ  similar  arrays  of  outcome-focused  quality  assurance 
mechanisms,  but  all  have  evolved  into  somewhat  unique  mechanisms.  This  section 
begins  by  explaining  the  service  system  context  in  each  state,  in  particular  how  quality 
assurance  is  managed.  It  then  identifies  the  major  tools  and  processes  the  states  are 
using  to  assess  consumer  outcomes,  and  discusses  the  relative  merits  of  each  type  of 
approach.  Common  themes  are  then  presented,  in  the  evolution  of  the  outcome- 
focused  systems  and  the  challenges  to  their  implementation.  Finally,  the  states  are 
compared  on  their  commitment  to  outcomes,  the  breadth  of  their  use  of  the  outcome- 
oriented  tools,  and  their  use  of  the  outcome  information. 

The  ensuing  analysis  is  based  on  a  review  of  relevant  quality  assurance  materials  in 
each  state  as  well  as  key  informant  interviews  in  each  state  (a  list  of  those  interviewed 
is  included  in  Appendix  E). 

It  is  important  to  note  that  as  these  QA  reforms  have  been  occurring,  states  have 
simultaneously  been  faced  with  other  ~  sometimes  countervailing  —  pressures  for 
change.  All  states  are  experiencing  budgetary  uncertainties,  and  many  feeling 
immediate  pressures  to  adopt  a  Managed  Care  approach  to  contain  costs  and  transfer 
decision-making  and  risk  to  local  levels.  Colorado  has  put  forth  a  "Blueprint  for 
Change/'  laying  out  a  possible  plan  for  greater  decentralized  authority  to  the 
Community-Centered  Boards;  Oklahoma  has  a  Governor's  Performance  Team  that  has 
just  completed  a  study  of  all  departments,  identifying  ways  to  reduce  service  costs. 
Massachusetts  is  facing  both  of  these  pressures,  in  the  Governor's  plan  to  restructure 
state  government  including  a  move  to  reliance  on  managed  care  organizations.  These 
challenges,  coupled  with  growing  waiting  lists,  will  continue  to  strain  that  ability  of 
states  ability  to  develop  and  maintain  high  quality  community  residential 
arrangements. 

State  Systems 

Five  of  the  seven  states  examined  in  this  study  place  responsibility  for  developmental 
disabilities  services  at  or  below  the  divisional  level  in  state  government.  In  Missouri, 
the  Division  of  Mental  Retardation  and  Developmental  Disabilities  (DMRDD)  is  located 
within  the  Department  of  Mental  Health;  in  Oklahoma,  the  Developmental  Disabilities 
Services  Division  (DDSD)  is  within  the  Department  of  Human  Services;  and  in  Utah, 
the  Division  of  Services  for  People  with  Disabilities  (DSPD)  lies  within  the  Department 
of  Human  Services.  In  Oregon  and  Colorado,  responsibility  sits  one  level  lower  — 
Colorado's  Developmental  Disabilities  Services  (DDS)  is  within  the  Office  of  Health 
and  Rehabilitative  Services  in  the  Department  of  Human  Services;  in  Oregon,  the  Office 
of  Developmental  Disabilities  Services  (ODDS)  is  located  within  the  Mental  Health  and 
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Developmental  Disabilities  Services  Division,  under  the  umbrella  Department  of 
Human  Resources.  New  York  and  Massachusetts  have  a  separate  department  for 
developmental  disabilities:  Massachusetts'  Department  of  Mental  Retardation  (DMR) 
and  New  York's  Office  of  Mental  Retardation  and  Developmental  Disabilities 
(OMRDD)  report  directly  to  the  governor.  New  York  has  the  further  distinction  of 
being  the  only  one  of  the  study  states  which  has  a  separate  Division  of  Quality 
Assurance  (DQ A),  within  the  OMRDD. 

Service  System  Configuration 

It  is  helpful  to  understand  the  service  system  context  in  which  quality  assurance  reform 
is  taking  place  —  how  many  individuals  are  currently  in  what  types  of  settings,  and 
what  degree  of  control  the  state  DD  offices  have  over  those  various  residential 
arrangements  ~  in  order  to  more  fully  understand  the  pressures  states  may  feel  as  they 
downsize  large  ICFs/MR  and  struggle  to  maintain  quality  in  increasingly  diverse 
settings.  Table  V  1  shows  distribution  of  consumers  receiving  residential  services  in 
each  state,  in  terms  of  total  numbers  and  percentage  of  the  state's  residential  consumers 
(numbers  are  in  some  case  approximations). 
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Table  1-8 

STATE  RESIDENTIAL  SERVICES  POPULATION,  1995 


STATE 

Residential 
Consumers 

ICF/MR 
Consumers 

Public 

ICF/MR 

Consumers 

Community- 
Based 
ICFs/MR 
Consumers 

Consumers 
in 

Community 
Settings  <3 

Consumers 
in 

Community 
Settings  4+ 

CO 

4000 

245 
(6%) 

175 

70 

2400 
(60%) 

1200 
(30%) 

MA 

8800 

1900 
(22%) 

1900 

None 

6900* 
(78%*) 

6900* 
(78%*) 

MO 

6500 

1516 
(23%) 

1329 

187 

3000 
(46%) 

2000 
(31%) 

NY 

31219 

8803 
(28%) 

494 

8309 

7315 
(23%) 

15101 
(48%) 

OK 

4000 

2375 
(59%) 

575 

1700 

975 
(24%) 

675 
(17%) 

OR 

4000 

470 
(12%) 

470 

None 

1000 
(25%) 

2500 
(63%) 

UT 

2100 

864 
(41%) 

359 

505 

800 
(38%) 

434 
(21%) 

The  seven  states  vary  significantly  in  the  extent  of  their  reliance  on  ICFs/MR,  from 
minor  usage  in  Colorado  and  Oregon  to  substantial  reliance  in  Oklahoma  and  New 
York.  Among  community  residential  settings,  all  the  states  except  Oregon  utilize 
smaller  settings  (of  three  people  or  less)  more  frequently  than  they  do  the  larger  group- 
home  settings,  reflecting  their  commitment  to  more  individualized  living 
arrangements. 
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A  critical  factor  affecting  the  size  and  the  quality  of  the  ICF/MR  population  in  each 
state  is  where  responsibility  lies  for  ICF/MR  facilities.  All  state  DD  offices  administer 
their  public  institutions,  except  Colorado.  Private  ICFs/MR  tend  to  be  more 
independent.  Of  the  five  states  that  have  such  facilities,  two  DD  offices  do  not  have 
jurisdiction  over  the  private  system.  In  Utah  and  Oklahoma,  the  private  ICFs/MR  are 
under  the  jurisdiction  of  a  separate  department,  thus  governed  by  separate  quality 
assurance  standards.  Key  informants  in  these  agencies  were  also  interviewed.  The 
only  guaranteed  oversight  is  the  ICF/MR  certification  process,  making  that  process 
thus  even  more  important  in  moving  ICFs/MR  toward  consumer  outcomes. 

Quality  Assurance  Capability 

The  state  DD  offices  have  responsibility  for  assuring  quality  in  the  residential  settings 
they  administer;  they  also  usually  carry  responsibility  for  state  monitoring 
responsibilities,  sometimes  licensing  authority  (Oregon),  and  some  amount  of  training 
and  technical  assistance  capability  (Oregon,  Oklahoma,  Utah,  Wyoming).  Staffing 
levels  have  influenced  the  range  of  tasks  the  DD  office  can  shoulder  —  Colorado's  five 
Program  Quality  staff  are  no  match  for  New  York's  surveyor  staff  of  over  90  or 
Massachusetts'  more  than  50  QUEST  staff,  hence  Colorado  relies  much  more  heavily  on 
regional  staff  to  monitor  and  on  providers  to  self-evaluate. 

Primary  Outcome-Focused  QA  Mechanisms 

All  the  study  states  have  one  or  more  quality  assurance  mechanisms  that  assess  the 
selected  outcomes.  Table  1-9  highlights  the  characteristics  of  these  approaches. 
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Table  1-9 

OUTCOME-FOCUSED  QA  MECHANISMS 


STATE 

MECHANISM 

YEAR 
BEGAN 

PRIMARY 
RESPONSIBILITY 

USED  IN 
ICF/MR? 

DESCRIPTION 

CO 

COPAR 

1986 

DDS,  private  firm 

Yes 

Longitudinal  evaluation  of  service  outcomes  based  on 
interviews  with  consumers  and  primary  support  staff 

MA 

QUEST 

1993 

DMR 

No 

Enhancement-oriented  site  survey  focused  on  individual 
consumers 

MO 

Certification 
survey 

Piloted 
1995 

DMRDD 

No 

Enhancement-oriented  site  survey  by  diverse  team, 
focused  on  agency  as  a  whole 

MOAIDD 

Pre-pilot 

MOAIDD 

No 

Citizen  monitoring  visits  done  in  alternate  years  of 
certification  survey 

NY 

COMPASS 

Piloted 
1993-94 

DQA 

Optiona 

Quality  improvement  system  includes  management 
planning,  self-survey,  and  outcomes  review 

CARE 

1993 

DD  Council 

No 

On-site  review  of  residential  programs  (observation  and 
interviews)  by  consumers,  families  and  advocates,  in 
programs  that  volunteer 

OK 

QA  Survey 

1990 

DDSD 

No 

On-site  assessment  of  agency  compliance  with  AC 
standards  and  DDSD  contract  standards,  done  by  DDSD 
staff,  including  observation,  interviewing,  and  records 

OSU  Consumer 
Assessment 

1990 

OSU 

Yes 

Longitudinal  assessment  of  individual  consumers, 
including  interview  with  consumer  and  primary  support 
person,  and  home  observation 

AIM 

1991 

Tulsa  ARC 

No 

Citizen  monitoring  of  community  residences  done  by 
local  ARC  parents  and  consumers,  including  interview 
and  observation 

OR 

ROS 

1994 

(1985 

VOIS) 

ODDS,  U  of  O 

No 

Individual-based  outcome  monitoring  system,  including 
assessment,  service  planning,  and  implementation, 
completed  by  direct  care  staff 

AIM 

1991 

Oregon  ARC 

No 

Citizen  monitoring  of  selected  community  residential 
settings  done  by  local  ARC  volunteers  (parents, 
consumers,  and  other  citizens) 

UT 

QE  Outcomes 
Survey 

1991 

DSPD 

No 

On-site  assessment  of  agency  compliance  with  outcome 
standards,  based  entirely  on  observation  and  interview 
of  individual  consumers 

Consumer 
survey 

1995 

DSPD 

No 

Consumer  interview  used  as  supplement  to  QE  survey 

Volunteer 
Monitoring 

1980 
(USDC) 
1990 
(com) 

MRAU  initially, 
now  DSPD 

Yes 

Citizen  monitoring  in  all  residential  settings  by  parents, 
including  observation,  consumer  interviews,  and  parent 
telephone  interviews 
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Table  1-9  reveals  that  most  of  the  study  states  have  more  than  one  outcomes- 
focused  QA  mechanism.  The  most  common  types  of  processes  include 
certification-type  surveys,  in  four  states  (more  states  have  such  surveys,  but  not  all 
are  strongly  outcome-oriented);  consumer  survey  approaches,  in  five  states;  and 
citizen  monitoring  efforts,  in  five  states.  One  state,  New  York,  has  a 
comprehensive  quality  improvement  system,  although  other  states  have  less 
extensive  QA  techniques  addressing  the  same  issues.  Most  of  these  tools  are  not 
used  in  the  ICF/MR  facilities,  and  most  are  carried  out  by  the  state  DD  offices 
(citizen  monitoring  being  the  biggest  exception).  The  outcomes  focus  is  fairly 
new  in  all  the  states,  with  only  COPAR  and  the  ROS  predecessor,  VOIS, 
beginning  prior  to  the  1990/s. 

The  site  visits  revealed  that  the  states  have  quite  varied  reliance  on  outcome- 
oriented  QA  tools.  All  the  outcome  tools  are  focused  in  some  way  on  improving 
outcomes  of  consumers,  and  all  assess  the  process  of  reaching  the  desired  outcomes, 
but  not  all  of  them  actually  assess  the  achievement  of  the  outcomes.  For  example, 
Utah's  QA  outcomes  survey  measures  the  extent  to  which  "the  individual  is 
encouraged  and  assisted  to  have  friends  in  the  community  with  whom  he/she 
does  not  reside,"  but  it  does  not  assess  whether  such  friendships  are  formed; 

Another  important  difference  across  the  study  states  is  the  degree  of 
commitment  by  the  state  DD  office  to  promoting  outcome-focused  quality 
assurance  as  a  preferred  approach  over  more  traditional  regulatory  compliance 
techniques.  Several  of  the  states  have  embraced  outcome-oriented  QA  over  any 
other  approaches  (e.g.,  Colorado,  Oklahoma,  and  New  York).  By  contrast, 
Oregon,  while  committing  significant  resources  to  outcome-focused  QA,  has  not 
adjusted  its  licensing  approach  to  support  agencies  exploring  innovative  ways  to 
achieve  consumer  outcomes,  and  it  has  not  expanded  the  use  of  AIM  to  all 
community  residential  settings. 

Certification  Surveys 

Several  of  the  states  have  based  their  program  survey  processes  on  AC  or  CARF 
standards:  Oklahoma  currently  uses  AC  process  standards  and  Utah  has 
borrowed  heavily  from  ACD  to  build  its  outcomes  survey.  Other  states  have 
created  their  own  outcome-oriented  survey  processes  (Colorado,  New  York, 
Massachusetts)  have  modified  their  licensing  standards  accordingly.  The 
Missouri  and  Massachusetts  processes  are  very  labor  intensive  —  Missouri's 
certification  process  can  take  from  2  days  to  2  weeks  to  complete,  QUEST  from 
one  week  to  ten  weeks. 
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Self-assessment 


Self  assessment  is  an  alternative  to  state  agency  monitoring  in  only  two  states, 
Colorado  and  New  York,  and  only  for  selected  agencies  (in  New  York,  all 
COMPASS  participants).  Several  of  the  other  study  states  have  self-assessment 
processes,  but  these  do  not  substitute  for  an  on-site  visit  by  the  state  DD  office 
team. 

Deeming 

Deeming  providers  as  meeting  state  standards  when  they  receive  national 
accreditation  is  a  policy  in  only  one  state,  Missouri.  Oklahoma  does  recognize 
national  accreditation,  waiving  the  QA  survey  for  agencies  who  are  accredited 
by  AC  or  CARF.  Utah  is  encouraging  providers  to  pursue  AC  accreditation,  by 
supporting  a  Quality  Assurance  Consortium  where  agencies  discuss  their  efforts 
toward  accreditation. 

Consumer  Survey  Approaches 

Several  of  the  study  states  have  developed  consumer  survey  tools  to  assess 
consumer  outcomes.  University-based  researchers  in  Colorado,  Oklahoma, 
Oregon  and  Utah.  The  Oklahoma,  Colorado,  and  Oregon  tools  are  used 
longitudinally,  and  have  been  generating  data  for  several  years,  providing  an 
important  resource  for  studying  effectiveness.  A  key  differentiating 
characteristic  among  the  consumer  survey  instruments  is  where  responsibility 
for  measurement  lies.  An  external  entity  conducts  the  surveys  in  Oklahoma  and 
Colorado,  and  the  Oklahoma  survey  is  also  analyzed  and  refined  by  the  outside 
group  (OSU);  data-gathering  is  provider-based  in  Oregon  and  Wyoming,  the 
latter  allowing  the  providers  to  choose  their  own  tool;  only  in  Utah  are  the 
consumer  surveys  handled  by  DD  office  staff. 

The  consumer  survey  tools  are  generally  used  only  in  community  residential 
settings,  although  Colorado's  COPAR  and  Oklahoma's  Longitudinal  Survey  are 
done  with  people  in  all  settings.  In  nearly  all  the  states,  the  public  ICFs/MR  are 
exploring  ways  to  use  a  modified  version  of  the  state's  consumer  survey  tool,  so 
such  practices  will  likely  become  more  common  in  the  future. 

External  Volunteer  Monitoring 

Five  of  the  study  states  use  some  form  of  external  volunteer  monitoring, 
Missouri,  Oregon,  Oklahoma,  New  York  and  Utah.  The  citizen  monitoring 
approach,  while  considered  valuable  by  key  informants  in  the  states  visited,  has 
received  varying  levels  of  financial  and  official  support,  and  occupies  a 
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somewhat  precarious  position  in  state  QA  systems.  This  is  in  part  because  of  the 
difficulty  in  mounting  a  system  that  relies  on  volunteers,  in  part  because  of  a 
lack  of  stable  funding,  and  in  part  because  of  an  ambivalence  in  these  systems 
regarding  the  authority  that  should  be  vested  in  citizen  reviews.  The  experience 
in  these  states  suggests  that  to  survive,  citizen  monitoring  requires  advocacy 
sponsorship,  paid  coordinating  staff,  a  commitment  to  the  importance  of  citizen 
monitoring,  stable  funding,  and  systematic  training  of  volunteers. 

Continuous  Quality  Improvement  Mechanisms 

Focused  attention  to  overall  organizational  improvement,  as  exemplified  in 
continuous  quality  improvement  efforts  (CQI),  varies  considerably  among  the 
study  states,  although  it  is  recognized  as  a  crucial  process  in  improving 
consumer  outcomes,  especially  as  quality  assurance  responsibilities  are 
increasingly  delegated  to  regional  DD  offices  and  to  providers  themselves. 
Missouri  and  New  York  have  made  the  most  dramatic  commitment  to  CQI 
approaches  and  Missouri  similarly  puts  organizational  change  at  the  heart  of  the 
state-sponsored  certification  process.  Oregon  also  devotes  specific  resources  to 
fostering  organizational  change,  but  such  activities  are  not  mandated.  Other 
states  do  not  address  CQI  per  se,  but  have  adopted  much  of  the  spirit  of  the  CQI 
movement,  making  their  QA  surveys  noticeably  less  punitive. 

Health  and  Safety  Safeguards 

All  the  study  states  have  taken  the  movement  toward  consumer  outcomes 
seriously,  and  have  made  modifications  in  their  approaches  to  assuring  basic 
health  and  safety,  in  community  residences  as  well  as  in  ICFs/MR.  Like  the 
changes  in  ICF/MR  process,  most  of  the  study  states'  own  minimal  health  and 
safety  surveys  have  become  less  targeted  on  documentation  and  more  focused 
on  observation  and  interviews,  both  with  staff/primary  support  person  and 
consumers.  This  shift  has  not  reduced  attention  to  safeguarding  health  and 
safety;  rather,  it  has  made  the  efforts  more  clearly  targeted  on  specific  risk 
situations.  Several  states  have  added  "Red  Flag"  checklists  to  other  of  their  QA 
mechanisms  (e.g.  Oklahoma  AIM,  Utah's  QE  survey). 

The  incident  reporting  process  appears  to  be  a  weak  link  in  many  states,  with 
little  oversight  at  the  state  level  and  varied  attention  given  by  local  officials;  and 
some  states  are  swamped  with  insignificant  incidents,  making  them  unable  to 
respond  promptly  and  thoroughly  to  serious  incidents,  or  to  step  back  and 
identify  underlying  systemic  problems  (e.g.  staff  training,  supervision, 
environment).  New  York  is  the  exception,  where  DQA  staff  monitor  quarterly 
trend  reports  from  each  local  office,  and  follow  up  when  unusual  patterns 
appear. 
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With  respect  to  the  two  accreditation  mechanisms  reviewed,  the  ACD 
Performance  Measures  for  Organizations  include  health,  safety  and  welfare 
measures.  Within  the  CARF  administrative  review  of  the  provider  agency, 
"rights,  health  and  safety"  are  combined  as  one  of  eight  accreditation  criteria  for 
which  the  provider  must  show  it  has  documented  policies. 

The  States'  Shift  To  Outcomes-Focused  QA 
Making  the  Shift:  Impetus,  Key  Factors 

The  common  factors  influencing  the  study  states'  shift  toward  outcome-focused 
quality  assurance  were:  (a)  lawsuits  targeted  at  the  large  state  institutions 
(Oklahoma,  Oregon,  Massachusetts,  Utah,  and  Colorado);  (b)  leadership  and 
vision  in  the  state  DD  office  (Colorado  and  Oklahoma),  and  (c)  availability  of 
new  service  monies  through  the  HCBS  waiver  (Missouri).  In  the  states  selected, 
there  was  evidence  of  a  growing  consensus  regarding  the  importance  of  having 
outcome-focused  QA,  and  increasing  attempts  to  involve  all  key  stakeholders  in 
collaborative  efforts  to  direct  QA  and  systems  reforms. 

Another  key  factor,  cited  by  state  officials  in  Oklahoma,  was  the  a  recognition  of 
the  evolutionary  nature  of  the  shift  which  had  to  proceed  through  the  process- 
focused  phase,  as  a  foundation  for  a  focus  on  outcomes- 
Implementation  Challenges 

The  major  challenge  to  the  implementation  of  an  outcome-focused  quality 
assurance  system  is  changing  the  attitudes  and  expectations  of  providers  as  well 
as  surveyors  who  are  still  conditioned  to  regulatory  compliance  and  professional 
decision-making.  This  was  particularly  true  in  Missouri,  New  York  and 
Massachusetts.  Thus,  the  process  of  instituting  such  a  system  must  come  to  grips 
with  organizational  development  issues  in  order  to  ensure  a  common 
understanding  of  the  assumptions  that  underlie  an  outcome-based  system. 

Several  expected  challenges  did  not  materialize  as  significant  issues  in  the  study 
states.  One  anticipated  concern  was  that  states,  in  their  rush  to  outcomes,  would 
lose  control  over  basic  health  and  safety.  None  of  the  states  abandoned  their 
focus  on  health  and  safety,  but,  rather,  balanced  it  with  the  new/expanded  look 
at  consumer  outcomes.  Another  expected  problem  was  that  states  would  lose 
the  support  of  key  stakeholders.  To  the  contrary,  state  DD  officials  focused  on 
keeping  in  good  communication  with  the  field;  they  included  other  stakeholders 
in  planning  and  design  efforts;  and  they  lessened  their  punitive  and  prescriptive 
role  (especially  in  Oklahoma  and  Colorado).  A  third  anticipated  difficulty  was 
the  role  transition  for  QA  surveyors. 
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Aspects  Of  Outcomes-Focused  QA 
Key  Outcomes  and  Outcome  Measures 

Table  1-10  illustrates  the  distribution  of  the  HCFA  selected  outcomes  among  the 
QA  systems  in  the  seven  study  states. 


Table  1-10 

HCFA  SELECTED  OUTCOMES  IN  STATE  QA  INSTRUMENTS 


QA 

Component 
HCFA 

V/UKUIIK 

CO: 
COPAR 

MA: 
QUEST 

MO: 
Certification 
survey 

MO: 
MOAIDD 

NY: 
COMPASS 

NY: 
CARE 

OK: 
Consumer 
Assessment 

OK: 
AIM 

OR: 
ROS 

OR: 
AIM 

UT: 
QE& 
Consumei 
Satisfy. 
Survey 

Self- 

determination 

X 

X 

X 

X 

X 

X 

V 
A 

X 

X 

X 

X 

Dignity 

X 

X 

X 

X 

X 

X 

X 

Health  and 
Safety 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Consumer 
Satisfaction 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Individualism 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Personal 
Growth 

X 

X 

X 

X 

X 

X 

X 

X 

Community 
Inclusion 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Relationships/ 
Social 

Connections 

X 

X 

X 

X 

X 

X 

X 

X 

X 

The  near-complete  attention  to  the  eight  valued  outcomes  evident  in  Table  1-10 
masks  what  appears  to  be  distinct  stages  in  the  evolution  of  states7  ability  to 
focus  clearly  on  consumer  outcomes.  Respondents  in  several  states  commented 
that  one  cannot  look  only  at  outcomes,  but  has  to  continue  to  attend  to  process. 
In  states  where  the  vision  of  systems  reform  and  the  commitment  to  an  outcomes 
focus  have  been  strong,  and  where  a  broad  array  of  stakeholders  share  that 
vision  and  commitment  (Colorado,  Missouri,  and  Oklahoma),  the  move  to 
outcomes  appears  to  have  been  the  steadiest  -  fairly  consistent  across  programs 
and  over  time.  Where  the  state  has  pushed  for  change  without  the  providers 
being  fully  willing  partners  (Oregon,  Utah,  and  Massachusetts),  one  does  not  get 
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the  same  sense  of  a  system-wide  transformation  to  outcomes.  These  states  still 
need  to  pay  close  attention  to  process,  and  especially  to  minimal  standards  of 
health  and  safety,  because  providers  do  not  necessarily  willingly  make  the 
change.  Some  observers  may  argue  it  has  to  do  with  the  level  of  disability  of 
people  living  in  the  community,  but  observation  does  not  support  that  thesis. 
Wyoming  stands  alone  in  a  different  stage  of  evolution,  where  some  of  the 
providers  have  led  the  change,  and  now  do  not  want  the  state  to  impose  any 
mandates;  the  result  is  that  DDD  has  no  one  outcome  tool  that  has  been  used 
consistently  over  time  or  across  providers,  making  it  less  able  to  assess  overall 
system  performance  in  an  outcomes  context.  The  imminent  adoption  of  CARFs 
outcome  standards  may  remedy  that  problem. 

Applicability  To  Various  Settings  and  Various  Populations 

As  a  general  rule,  the  DD  offices  in  the  study  states  have  not  applied  their 
outcome-focused  tools  in  ICF/MR  facilities,  usually  because  either  the  bulk  of 
the  ICFs/MR  were  not  under  the  jurisdiction  of  the  state  DD  office  (Utah  and 
Oklahoma),  or  the  ICFs/MR  were  seen  as  appropriately  and  sufficiently 
regulated  by  HCFA.  The  states  tended  to  give  priority  to  designing  QA 
mechanisms  for  settings  where  no  other  agency  was  providing  oversight 

People  With  the  Most  Severe  Disabilities.  State  quality  assurance  officials  in  the 
seven  states  see  consumer-centered  outcomes  as  being  fully  applicable  to  people 
with  the  most  severe  disabilities.  In  particular,  these  respondents  emphasized 
that  face-to-face  interviewing  with  consumers  is  crucial  for  all  consumers, 
especially  in  assessing  the  outcomes  of  satisfaction,  social  relationships,  dignity, 
and  choice.  The  only  consistent  reservation  expressed  concerned  people  with 
significant  medical  needs,  the  argument  being  that  more  careful  attention  to 
health  and  safety  is  necessary  for  this  group  of  people. 

Colorado  and  Oklahoma  have  made  particular  efforts  to  make  their  outcome- 
focused  tools  useful  with  people  with  the  most  severe  communication 
difficulties,  Oklahoma  by  using  interviewers  with  sign  language  and  receptive 
skills,  as  well  as  a  picture  book,  and  Colorado  by  revisions  to  the  COPAR 
instrument.  In  addition,  both  states'  consumer  survey  processes  supplement  the 
consumer  information  with  information  from  primary  support  people.  Oregon 
relies  primarily  on  direct  care  staff  to  record  the  relevant  information  on  an 
ongoing  basis,  with  little  direct  consumer  interviewing  about  outcomes. 

New  York  quality  assurance  officials  emphasize  the  need  for  special  training  for 
surveyors  in  applying  the  core  outcomes  to  people  with  severe  disabilities, 
especially  assessing  choice  and  satisfaction  for  people  with  severe 
communication  disabilities;  they  currently  make  use  of  secondary  respondents 
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(primary  caregivers),  and  some  agencies  have  trained  surveyors  to  read  non- 
verbal cues.  Missouri,  like  Colorado  and  Oklahoma,  has  successfully 
interviewed  individuals  living  in  waiver-funded  homes  who  have  severe 
disabilities. 

Quality  assurance  officials  in  New  York  also  noted  that  outcome  expectations 
may  need  to  be  somewhat  different  for  people  with  severe  disabilities,  especially 
in  the  areas  of  friendships  and  choice-making.  New  York  DQA  staff  highlighted 
the  distinction  between  outcomes  promoted  and  outcomes  achieved:  they  felt  it 
is  was  important  to  identify  situations  where  the  staff  provided  an  environment 
which  promoted  the  achievement  of  the  outcomes  but  where  the  outcomes  were 
not  fully  achieved  due  to  the  nature  of  the  individual's  disabilities  or  the  choices 
he/she  made.  Utah  has  developed  such  a  schema  in  its  scoring  of  the  QE  survey 
indicators. 

Institutions  and  Community-Based  ICFsJMR.  A  consensus  similar  to  that  around 
consumers  with  varying  levels  of  disability  emerged  in  considering  the 
applicability  of  outcomes  to  various  types  of  residential  settings.  In  general,  the 
outcomes  are  seen  by  state  quality  assurance  officials  as  being  applicable  to 
people  living  in  ICFs/MR,  although  they  were  quick  to  acknowledge  the 
differing  potential  for  outcomes  in  different  residential  models  (for  example, 
Oklahoma's  Assisted  Living  residents  receive  less  supports  and  generally  have 
less  money;  remote  institutional  campuses  face  considerably  greater  barriers  to 
achieving  community  inclusion). 

Active  treatment  is  one  area  where  the  compatibility  of  the  valued  outcomes 
with  current  ICF/MR  regulations  is  ambiguous.  Utah's  ICF/MR  administrators 
do  not  see  a  conflict  between  active  treatment  and  the  outcomes  standards.  They 
see  the  former  is  the  basics,  the  latter  is  more  creative,  and  they  say  it  is  easy  to 
marry  the  two,  with  active  treatment  as  the  foundation.  They  feel  supported  in 
this  view  by  the  HCFA  regional  office  training  they  have  received.  In  sharp 
contrast  is  Oregon's  experience,  where  ICF/MR  surveyors  are  having 
considerable  difficulty  seeing  the  institutions'  efforts  in  person-centered 
planning  as  meeting  active  treatment  requirements. 

It  appears  that  the  potential  problems  with  the  applicability  of  outcome 
standards  in  an  ICF/MR  environment  are  more  related  to  the  routinized  patterns 
and  social  isolation  of  the  institution  (the  "culture"),  rather  than  being 
inappropriate  indicators  for  people  with  severe  disabilities.  This  suggests  that  a 
precursor  to  implementing  outcome-focused  QA  in  the  ICFs/MR  should  be 
extensive  training  and  acculturation,  among  staff  and  surveyors  alike. 
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Finally,  with  respect  to  accreditation  mechanisms,  to  the  best  of  the  research 
team's  knowledge,  thus  far,  no  agency  supporting  more  than  200  individuals  in 
one  setting  has  received  accreditation  under  the  outcome  standards. 

The  Regulatory  Basis  for  the  Outcome  Focus 

Where  the  state  has  an  explicit  vision  and  mission  statement  in  legislation,  the 
outcomes  focus  has  a  firmer  basis;  this  is  true  for  Oklahoma,  Massachusetts  and 
Colorado,  and  somewhat  less  so  for  Utah.  Many  more  of  the  study  states  have 
some  or  all  of  the  outcome-oriented  QA  mechanisms  stipulated  in  consent 
decrees,  which  has  in  some  cases  been  a  springboard  for  using  the  tools  more 
broadly  than  simply  for  class  members  (especially  in  Oklahoma). 

Integration  Among  QA  Mechanisms 

Findings  from  outcome-oriented  surveys  have  not  been  widely  shared  in  the 
study  states,  nor  have  the  data  been  routinely  analyzed  in  conjunction  with  other 
quality  assurance  data  sets.  In  terms  of  information-sharing,  the  consumer 
survey  findings  are  shared  less  than  are  the  citizen  monitoring  or  certification 
survey  results.  Both  Colorado's  COPAR  and  Oklahoma's  consumer  assessment 
have  not  been  routinely  shared  with  case  managers  or  providers.  In  Oregon,  by 
contrast,  the  ROS  findings  are  promptly  fed  back  to  direct  care  staff,  to  help 
them  reflect  on  their  own  practice.  None  of  the  states  gives  the  consumer 
interview  information  to  licensing  or  certification  authorities,  although  such 
information  in  the  aggregate  could  offer  valuable  insight  into  provider 
performance.  Citizen  monitoring  reports  are  commonly  shared  with  other  QA 
authorities  as  well  as  with  providers.  WY  does  better  than  other  states  in 
sharing  information,  probably  because  it  is  a  smaller  system. 

Analysis  of  the  outcomes  data  has  similarly  been  fairly  limited.  Colorado  has 
done  extensive  analysis  of  the  COPAR  results,  both  longitudinal  and  cross- 
sectional,  but  with  no  formal  linkage  between  COPAR  and  other  data  sets. 
Oklahoma  has  followed  a  similar  pattern  with  its  longitudinal  consumer 
assessment  data.  Oregon  has  begun  to  do  some  longitudinal  analysis  of  the  ROS 
data,  but  nothing  has  been  done  with  AIM  results.  Wyoming  is  beginning  to 
consider  a  more  extensive  analytic  QA  function  through  the  UAP.  Utah  has 
done  limited  analysis  of  the  QE  outcome  surveys,  comparing  performance  across 
residential  settings  and  ICAP  levels. 

Many  of  the  states  make  use  of  the  outcome-focused  QA  information  for  policy 
purposes.  Colorado  sees  COPAR's  primary  function  as  influencing  state  policy, 
and  New  York's  COMPASS  has  a  comparable  program  evaluation  focus. 
Several  of  the  study  states  share  results  of  the  consumer  surveys  with  legislature 
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and /or  governor  -  for  example,  annual  reports  from  the  DD  offices  in  Utah, 
Colorado,  and  Oklahoma.  Several  states  distribute  QA  information  very  broadly 
-  Utah's  DSPD  has  a  Consumer  Report  on  Home  and  Community  based  Services,  to 
provide  information  to  consumers  to  assist  them  in  choosing  service  providers;  it 
indicates  how  each  agency  performed  on  the  QE  survey,  the  consumer 
satisfaction  surveys,  and  the  volunteer  monitoring  visits.  Oklahoma  attempted 
to  do  a  consumer  report  but  found  such  resistance  from  providers  that  they 
abandoned  it.  Wyoming  DDD  publishes  and  widely  disseminates  the  reports  of 
their  CARF-based  state  certification  visits. 

Several  states  are  attempting  to  improve  the  integration  of  QA  information  from 
various  sources,  so  that  it  becomes  more  useful  for  both  policy  and  practice.  The 
preferred  vehicle  for  these  efforts  is  collaborative  planning  groups: 

•  Oregon's  QA  Summit  group,  composed  of  representatives  of  all  parties 
engaged  in  some  QA  activity,  is  meeting  quarterly  to  develop  joint 
ownership  in  quality  assurance,  to  collectively  develop  strategies  for 
fixing  things,  and  to  build  a  process  to  integrate/link  data  from  the 
various  QA  mechanisms;  the  goal  is  to  identify  some  key  questions, 
benchmarks  of  quality,  and,  in  the  process,  stimulate  similar  efforts  at 
the  county  level  to  link  QA  information  and  assess  their  own 
performance. 

•  Utah's  Futures  Directions  Strategic  Planning  group,  composed  of 
representatives  of  all  key  stakeholders,  meets  annually  to  guide  DSPD  in 
its  services  and  QA  system.  The  group  reviews  the  prior  year's  action 
plans,  recommends  changes,  publishes  a  report,  and  celebrates 
accomplishments. 

Stakeholder  Involvement 

Citizen  involvement  is  the  core  of  the  volunteer  monitoring  processes.  All  of  the 
study  states  have  some  type  of  advisory  body  to  the  DD  office,  and  most  have 
advisory  bodies/steering  committees  for  specific  QA  components  (e.g. 
COMPASS,  COPAR).  Colorado's  agency-directed  surveys  can  include  non- 
professionals as  team  members,  and  Missouri's  certification  survey  teams 
include  consumers,  families,  and  providers. 

Themes  and  Recommendations 

Based  on  the  50  state  survey  and  the  state  site  visits,  there  are  some  general 
themes  that  are  important  to  HCFA's  considerations  regarding  future  regulatory 
directions. 
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Relevance  of  Outcome  Measures  to  Large  ICFs/MR 

With  some  exceptions,  the  outcome-based  mechanisms  that  were  reviewed  were  not 
uniformly  applied  to  people  who  live  in  large  ICFs/MR.  Where  they  were  applied,  as 
in  Massachusetts,  the  facility  did  not  score  favorably  compared  to  community-based 
residences,  although  the  process  was  seen  as  positive  for  those  who  worked  in  such 
facilities.  The  absence  of  these  facilities  from  the  state  initiatives  also  means  that  there  is 
not  much  information  gained  from  the  state  site  visits  regarding  the  use  of  the  measures 
on  people  living  in  large  institutions.  The  analysis  conducted  by  Temple  University 
(summarized  in  the  following  section)  as  part  of  this  project,  however,  does  show  that 
the  instruments  do  have  applicability  in  these  settings. 

It  is  therefore  recommended  that  any  change  in  HCFA's  current  regulations  regarding 
the  selected  outcomes  be  applied  across  ICF/MR  settings  regardless  of  size.  The 
current  absence  of  such  oversight  mechanisms  in  large  public  institutions  should  not 
justify  the  development  of  a  bifurcated  system  in  which  people  who  live  in  institutions 
are  not  subject  to  the  achievement  of  the  selected  outcomes. 

Increase  in  the  Use  of  Self-Assessments 

In  many  of  the  states,  reliance  on  self-assessments  was  increasing.  In  some  states,  like 
New  York  and  Colorado,  providers  chosen  to  carry  out  self-assessments  were  those 
with  high  scores  on  traditional  quality  assurance  reviews.  By  doing  a  self-assessment, 
providers  were  excused  from  these  regular  regulatory  reviews.  In  New  York,  follow-up 
evaluation  showed  that  where  agencies  did  self-assessments,  they  still  maintained 
compliance  with  formal  requirements  —  in  some  instances  at  higher  levels  than 
previously.  The  motivation  for  the  increased  use  of  self-evaluation  was  both  born  of 
necessity  (shrinking  resources  at  the  central  level),  as  well  as  a  commitment  to  quality 
enhancement.  Advocates  interviewed  in  Colorado  expressed  deep  concerns  about  this 
trend.  However,  providers  interviewed  in  New  York  and  Colorado  saw  the  self- 
assessment  as  a  positive  step  and  a  sign  of  increased  public/private  collaboration 
between  the  agency  and  the  state. 

Given  the  current  and  potential  limitations  on  state  quality  assurance  resources,  the  use 
of  self-assessment  in  conjunction  with  some  aspects  of  formal  monitoring  for  those 
providers  who  have  demonstrated  a  commitment  to  quality,  can  both  stretch  QA 
resources  and  reward  outstanding  providers.  However,  it  is  recommended  that  some 
threshold  health  and  safety  review  be  maintained. 
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Interviewing  People  with  Severe  Disabilities/Limited  Communication 

Virtually  all  of  the  outcome  systems  that  were  reviewed  involved  some  level  of 
consumer  response  regarding  outcomes  they  are  experiencing  and,  in  most  cases,  their 
satisfaction  with  services  and  supports.  All  of  the  outcome  measurement  systems 
involved  direct  contact  with  consumers  and  many  states  and  localities  employed 
additional  consumer  satisfaction  measures.  While  this  emergence  of  consumer-centered 
quality  assurance  is  admirable,  some  questions  did  emerge  regarding  the  use  of 
interviews  on  outcomes  for  people  with  significant  communication  limitations,  such  as 
their  choice-making,  opportunities  for  inclusion  and  relationships,  and  satisfaction. 
Some  questions  were  also  raised  about  the  numbers  of  individuals  that  could  respond 
to  quality  assurance  and  outcome  surveys.  In  Colorado,  the  estimates  were  about  25% 
and  key  informants  in  other  states  noted  similar  limitations.  In  some  states,  such  as 
New  York  proxy  respondents  (family  and  direct  support  staff)  were  used  to  complete 
the  information.  There  is  some  concern,  however,  that  such  partial  data  may  be  used  to 
justify  program  performance  in  the  absence  of  other  reviews  or  more  innovative 
approaches  to  obtaining  consumers'  perspectives  on  outcomes. 

This  finding  has  several  implications.  First,  as  noted  in  Report  #2  regarding  the  current 
ICF/MR  review  process,  surveyors  will  need  systematic  training  to  communicate  with 
people  with  severe  and  profound  disabilities  and  to  elicit  information  from  those  who 
are  closest  to  the  individual.  Secondly,  consumer  "satisfaction"  cannot  be  the  only 
measurement  of  quality  ~  consumer  survey  approaches  must  be  coupled  with  ongoing 
health  and  safety,  some  program  or  "process"  reviews,  and  the  use  of  objective 
performance  indicators  (see  the  report  with  respect  to  Task  B  of  this  project). 

Training  Surveyors  in  Outcome-Based  Methods 

Most  state  quality  assurance  officials  emphasized  the  importance  of  training  surveyors 
in  using  new  outcome-based  instruments  and  related  techniques,  in  particular 
interviews  and  observations.  In  Colorado,  the  state  program  agency  trains  surveyors 
hired  by  a  private  contractor,  while  the  New  York  program  office  trained  a  subgroup  of 
surveyors  in  the  COMPASS  program.  Surveyors  also  received  special  training  in 
Massachusetts  on  QUEST.  Training  was  considered  important  in  shifting  to  outcome- 
focused  quality  assurance  to  re-orient  surveyors  used  to  traditional  licensure  reviews,  to 
enhance  skills  in  obtaining  information  through  interviews  and  observation,  and  to 
improve  inter-rater  reliability.  It  also  has  been  suggested  that  many  ICF/MR  surveyors 
will  need  considerable  (re-)  training  in  order  to  shift  to  more  of  an  outcome  focus. 

With  respect  to  HCFA's  concerns,  this  finding  underscores  the  importance  of  the 
current  surveyor  training  initiative  as  well  as  the  maintenance  of  a  training  program 
directed  at  outcomes  in  the  future. 
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Outcome  Measurement  and  Direct  Support  Staff 

Outcome,  consumer-focused  approaches  were  very  popular  with  direct  support  staff 
who  saw  them  as  a  validation  of  the  importance  of  their  work  and  the  people  that  they 
serve.  Interestingly,  they  were  more  positive  in  most  instances  than  the  administrators 
of  programs. 

This  finding  strongly  suggests  that  any  outcome-focused  quality  assurance  mechanism 
should  include  organizational  development  strategies  that  assist  direct  support  and 
other  staff  to  understand  the  implications  of  an  outcome  focus  for  their  agency  in 
contrast  to  more  process-oriented  approaches.  Training  would  include  the  importance 
of  respecting  the  preferences  and  goals  of  people  with  mental  retardation,  the  ways  in 
which  staff  performance  reviews  would  be  tied  to  reconfigured  expectations  based  on 
outcomes,  and  the  ways  in  which  outcomes  would  be  reported  and  represented. 

Importance  of  Organizational  Development 

Many  of  the  quality  assurance  systems  reviewed  included  either  an  implicit  or  an 
explicit  emphasis  on  the  organizational  development  of  providers.  Given  the 
significant  changes  that  many  organizations  must  initiate  in  order  to  manage  services  in 
a  more  "outcome  oriented"  fashion,  the  inclusion  of  an  organizational  development 
strategy  would  appear  to  be  an  important  ingredient.  The  most  explicit  example  of  an 
organizational  development  approach  combined  with  an  outcome  focus  was  the 
COMPASS  program  in  New  York.  To  become  an  COMPASS  agency,  programs  were 
required  to  develop  strategic  plans,  internal  QA  systems,  and  a  consumer-focused 
approach. 

This  finding  suggests  that  HCFA  should  consider  revising  the  Conditions  of 
Participation  in  the  ICF/MR  program  to  include  an  explicit  requirement  for 
organizational  development  aimed  at  a  transition  to  an  outcome  focus. 

Role  of  Citizen  Monitoring 

Citizen  monitoring  of  programs  for  people  with  mental  retardation  was  a  facet  of  many 
of  the  quality  assurance  systems  reviewed.  Citizen  monitoring  has  the  benefit  of 
extending  scarce  monitoring  resources  and  providing  multiple  perspectives  including 
families  and  people  with  disabilities.  Citizen  monitoring,  however,  requires  significant 
coordination  and  support  to  ensure  the  integrity  of  the  process.  In  some  of  the  states 
reviewed,  the  strength  of  citizen  monitoring  had  waxed  and  waned  over  the  years  and 
while  it  is  clearly  an  important  element  of  a  quality  assurance  system,  it  has  been 
difficult  to  elevate  it  to  a  more  central  role  than  it  currently  occupies. 
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It  is  recommended  that  HCFA  consider,  as  a  condition  of  participation  in  the  ICF/MR 
program,  the  presence  of  an  external  monitoring  entity  ~  preferably  that  includes 
citizens  and  people  with  disabilities. 

Health  and  Safety  Protections 

The  conversion  of  ICFs/MR  and  the  reduction  in  the  size  of  residential  arrangements 
has  meant  that  ICF/MR  regulations  and  state  licensing  requirements  aimed  at  larger 
settings  (e.g.,  5  or  more  beds)  are  not  relevant  to  many  small  residential  arrangements. 
In  Colorado,  for  instance,  licensing  does  not  apply  to  residences  under  four  beds. 
Further,  without  ICF/MR  regulations,  there  may  not  be  any  requirements  that 
incidence  and  abuse  reports  get  communicated  to  central  authorities.  While  the  decline 
of  potentially  unnecessary  regulation  is  positive,  there  were  informants  who  expressed 
some  anxiety  about  the  loss  of  some  basic  protections.  Throughout  most  of  the  states 
visited,  there  were  efforts  to  "streamline"  regulations  and  to  reduce  their  complexity. 
However,  it  should  also  be  noted  that  in  the  states  visited,  health  and  safety  issues  were 
either  included  in  the  outcome  monitoring  mechanism,  or  were  part  of  parallel  licensing 
and  certification  procedures. 

To  the  extent  that  this  is  a  federal  issue,  it  affects  the  way  in  which  the  federal  waiver 
program  is  implemented  since  community  residences  are  increasingly  supported  by 
waiver  funding.  It  is  suggested  that  joint  discussions  be  held  between  federal  waiver 
and  ICF/MR  officials  to  discuss  a  joint  strategy  to  ensure  that  valuable  protections  are 
not  diminished. 

Importance  of  Stakeholder  Consensus 

An  important  ingredient  in  the  implementation  of  the  outcome  structures  reviewed  for 
this  project  was  the  emphasis  placed  on  the  development  of  consensus  among  all  key 
stakeholders  including  people  with  disabilities,  families,  providers  and  policy  makers. 
In  Missouri,  for  instance,  the  development  of  that  state's  new  outcome-based 
certification  process  involved  a  wide  range  of  stakeholders  from  around  the  state.  In 
New  York,  the  state  QA  officials  worked  with  a  consumer  board  to  implement  the 
COMPASS  approach.  In  other  states,  stakeholder  groups  have  been  convened  to 
review  the  information  gained  from  the  outcome  monitoring  process. 

It  is  recommended  that  HCFA  signal  its  concern  from  broad  based  participation  in 
quality  assurance  plans  in  any  future  regulations. 
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Growing  Use  of  Accreditation 


Some  states  are  using  accreditation  as  the  core  or  an  alternative  to  their  quality 
assurance  process.  Many  other  states,  as  they  contemplate  managed  care,  are  also 
considering  accreditation  as  a  means  of  certifying  eligible  agencies.  Both  accreditation 
schemes,  ACD  and  CARF,  have  moved  to  an  outcome  focus.  The  potential  expansion  of 
the  "privatization"  of  quality  assurance  should  be  an  important  consideration  in  any 
future  HCFA  plans  for  ICFs/MR. 

Given  the  potential  expansion  of  accreditation  as  a  means  of  ensuring  program  quality, 
HCFA  should  consider  the  possibility  of  allowing  ICFs/MR  to  substitute  accreditation 
compliance  with  the  explicitly  programmatic  aspects  of  federal  standards. 
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Objective  3:  Review  of  the  reliability  and  validity  of  selected  quality  assurance 

instruments 

Summary  of  Findings 
Face  Validity 

Data  collection  efforts  in  the  seven  states  and  the  two  accreditation  agencies  yielded 
some  very  promising  results.  In  addition  to  having  the  attribute  of  face  validity,  which 
was  evident  in  the  development  of  the  instruments  used  to  assess  outcomes,  many  of 
the  instruments  proved  to  be  reliable  and  valid. 

Reliability 

The  reliability  of  the  instruments  assessed  overall  was  favorable.  In  five  of  the  nine 
states  and  organizations  reliability  was  found  to  be  well  within  acceptable  ranges.  The 
states  included:  Colorado,  Massachusetts,  Missouri,  Oklahoma,  and  Oregon.  In  Utah, 
where  a  reliability  study  was  conducted,  the  reliability  rate  was  unacceptable.  This  was 
thought  to  be  related  to  the  lack  of  experience  of  one  individual.  Utah  plans  to  redo  its 
reliability  study  once  this  individual  has  had  additional  training.  With  regard  to  New 
York,  the  instrument  was  found  to  be  unreliable.  Interestingly,  it  has  been  determined 
that  the  instrument  will  no  longer  be  used.  In  terms  of  the  accrediting  agencies,  the 
Accreditation  Council  is  currently  in  the  process  of  completing  an  extensive  reliability 
study.  It  would  be  our  recommendation  that  CARF  undergo  similar  efforts. 

Concurrent  Validity 

In  terms  of  concurrent  validity,  the  results  were  not  as  promising.  Concurrent  validity 
was  found  to  be  acceptable  for  three  of  the  nine  states  and  agencies  (CO,  OK,  OR) 
marginal  for  5  of  the  states  and  agencies  (MA,  MO,  UT,  AC,  CARF)  and  unacceptable  in 
one  state  (NY).  After  careful  analysis  of  the  data,  it  became  clear  that  there  were  several 
possible  explanations  for  the  weakness  of  these  findings.  First,  in  many  cases  where 
concurrent  data  using  the  Temple  BDS  were  collected  long  after  the  State  and  Agency 
data  had  been  collected,  it  is  possible  that  things  had  really  changed.  In  the  case  of  AC, 
for  example,  the  Temple  BDS  data  were  collected  about  one  year  after  the  Accreditation 
Survey  had  been  completed.  There  is  no  doubt  that  within  one  year,  things  had 
changed.  Second,  in  many  cases  the  reports  that  were  used  to  produce  the  concurrent 
data  from  the  states  and  agencies  were  site  data  rather  than  individual  data.  In  that 
situation,  the  score  of  the  agency  was  attributed  to  each  individual  and  then  compared 
to  data  that  were  collected  at  the  individual.  It  is  for  this  reason,  we  believe,  that  the 
states  where  the  concurrent  validity  analyses  produced  the  strongest  results  were  the 
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states  where  individual  data  are  collected.  Future  efforts  should  be  targeted  toward 
obtaining  enough  site  data  to  truly  look  at  a  large  number  of  agencies  and  sites. 

Applicability 

Applicability  was  acceptable  in  five  of  the  nine  states  and  agencies  (CO,  MA,  NY,  OK, 
CARF),  marginal  in  one  agency  (AC),  and  unacceptable  in  three  states  (MO,  OR,  UT). 
The  lack  of  applicability  in  the  three  states  is  not  related  to  inability  to  be  applied  in 
larger  settings,  but  rather  because  those  three  states  have  chosen  to  use  the  outcome 
assessments  only  in  the  community  programs.  In  terms  of  AC,  no  facility  has  been 
accredited  that  is  larger  than  120  individuals. 

In  terms  of  discriminant  validity,  acceptable  relationships  were  found  in  five  of  the  nine 
states  and  agencies  (CO,  MO,  NY,  OK,  UT),  and  in  the  remaining  states  and  agencies 
there  were  insufficient  data  to  complete  the  analyses.  In  terms  of  the  two  accrediting 
agencies  and  the  State  of  Massachusetts,  because  scores  are  ascribed  to  agencies,  the 
discriminating  variables  have  nothing  to  predict.  In  other  words,  if  one  wanted  to 
assess  the  ability  of  level  of  mental  retardation  (4  levels)  to  predict  integration  and  there 
is  only  one  integration  score  for  everyone  in  the  facility  the  lack  of  variance  makes  such 
an  exercise  futile.  On  the  other  hand,  the  State  of  Oregon  maintains  no  demographic 
data  on  the  outcome  data  system;  therefore  there  were  no  predicting  variables  available 
for  analysis. 

Health  and  Safety  data  were  collected  in  all  nine  states  and  agencies.  In  most  states  and 
agencies,  the  Health  and  Safety  data  were  compared  with  the  states'  outcome  data.  In 
all  states,  comparisons  were  made  with  state  licensing  data  and  ICFs/MR  OSCAR  data. 
Comparisons  between  the  outcome  data  and  the  Health  and  Safety  data  yielded  only  a 
few  significant  correlations,  most  of  which  appear  to  be  spurious.  This  may  be  due  to 
the  fact  that  no  significant  relationship  between  outcomes  and  health  and  safety  was 
measured.  As  a  result,  it  is  the  study  team's  firm  belief  that  the  two  dimensions  be 
assessed  separately.  There  were  correlations  between  the  state  and  federal  deficiency 
data  and  some  of  the  health  and  safety  scales.  Overall,  our  conclusion  is  that  the  Health 
and  Safety  scales  were  not  highly  correlated  with  any  of  the  outcome  measures 
analyzed. 

The  results  of  the  various  analyses  are  summarized  in  Table  1-11  below. 
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Table  1-11 

SUMMARY  OF  PSYCHOMETRIC  PROPERTIES  OF  INSTRUMENTS 
AND  HEALTH  AND  SAFETY  ANALYSES 


CO 

Reliability 

Assessed  by  Temple, 
acceptable 

Concurrent 
Validity 

Assessed  by  Temple, 
acceptable 

Applicability 

Assessed  by  Temple, 
acceptable 

Discriminant 
Validity 

Assessed  by  Temple, 
acceptable 

Health 
&  Safety 

Assessed  by  Temple 

MA 

Assessed  by  MA, 
acceptable 

Assessed  by  Temple, 
marginal 

Assessed  by  MA 
acceptable 

Assessed  by  Temple, 
insufficient  data 

Assessed  by  Temple 

MO 

Assessed  by  MO, 
acceptable 

Assessed  by  Temple, 
marginal 

Unacceptable 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple 

NY 

Assessed  by  Temple, 
unacceptable 

Assessed  by  Temple, 
unacceptable 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple 

OK 

Assessed  by  OK, 
acceptable 

Same  scales, 
acceptable 

Assessed  by  OK, 
acceptable 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple 

OR 

Assessed  by  OR, 
acceptable 

Assessed  by  OR, 
acceptable 

Unacceptable 

Assessed  by  Temple, 
insufficient  data 

Assessed  by  Temple 

UT 

Assessed  by  UT, 
unacceptable 

Assessed  by  UT, 
marginal 

Unacceptable 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple 

AC 

Currently  being 
assessed  by  AC 

Assessed  by  Temple, 
marginal 

Assessed  by  AC, 
marginal 

Assessed  by  Temple, 
Temple 

insufficient  data 

Assessed  by 

CARF 

Insufficient  data 

Assessed  by  Temple, 
marginal 

Assessed  by  Temple, 
acceptable 

Assessed  by  Temple, 
Temple 

insufficient  data 

Assessed  by 

58 


Implications  and  Recommendations 

In  summary,  the  quantitative  portion  of  this  study  demonstrated  that  there  are  systems 
currently  in  use  in  the  states  that  in  addition  to  having  been  developed  over  long 
periods  of  time  with  significant  thought  and  input  from  various  stakeholders,  many  of 
the  systems  possess  strong  statistical  properties.  The  study  team  offers  the  following 
recommendations  regarding  the  implementation  of  outcome-based  quality  assurance 
systems: 

1)  All  systems  should  be  tested  for  reliability,  validity  and  applicability  across 
settings  and  across  individuals  with  a  variety  of  disabilities; 

2)  In  systems  where  scores  are  ascribed  to  facilities  rather  than  individuals,  great 
pains  should  be  taken  to  insure  that  the  facility  scores  adequately  represent  the 
experience  of  the  individuals  supported;  in  other  words,  greater  attention  needs 
to  be  paid  to  the  issue  of  appropriate  sample  size; 

3)  Health  and  safety  should  be  assessed  separately  from  the  assessment  of 
outcomes; 

4)  Stakeholders  must  decide  whether  the  outcome  systems  are  assessment 
systems,  technical  assistance  systems  or  a  combination  of  the  two  -  this  should  be 
clearly  articulated  from  the  outset 
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Objective  4:  Results  of  the  review  of  empirical  studies  of 
outcome  focused  quality  assurance 

Summary 

While  some  commonalties  exist  among  states  moving  to  incorporate  quality  of  life 
outcomes  into  their  evaluation  strategies,  it  is  clear  that  each  state  is  forging  its  own 
path.  This  may,  in  part,  be  attributed  to  the  plethora  of  "quality  of  life"  definitions  and 
measurement  approaches.  In  one  recent  study  (Hughes,  Hwang,  Kim,  Eisenman  & 
Killian,  1995)  reviewing  literature  from  the  past  two  decades,  44  distinctly  different 
definitions  of  quality  of  life  were  discovered.  While  the  fundamental  "dimensions"  or 
"components"  of  quality  of  life  parallel  those  utilized  in  this  project  study,  it  is 
important  to  note  that  they  encountered  over  1200  unique  measures  used  to  assess  these 
dimensions. 

The  areas  of  independence,  social  relationships,  community  integration,  self- 
determination/choice,  and  consumer  satisfaction  were  common  elements  for 
assessment  in  most  of  the  studies  identified.  Another  consistency  included  state  efforts 
to  make  comparisons  between  results  for  varying  service  options  (more  specifically 
residential  options),  and  comparisons  based  on  the  extent  of  disability  experienced  by 
service  recipients. 

The  differences,  however,  far  outweigh  the  similarities.  The  specific  indicators  or  target 
areas  of  quality  vary  in  nature.  Comparisons  between  residential  settings  are 
inconsistent  across  states  (e.g.,  institution  vs.  community,  ICF/MR  vs.  HCBW  funded 
settings,  small  vs.  large  community  settings).  And  although  use  of  surveys  and 
interview  protocols  are  generally  accepted  practice,  measurement  tools  and  approaches 
vary  significantly. 

In  summary,  very  little  data  and  reporting  exist  related  to  empirical  studies  of  quality 
outcomes,  and  among  that  which  does  exist,  there  is  minimal  consistency  across 
studies.  While  it  is  evident  that  states  are  moving  toward  outcome-driven  quality 
assurance  systems,  there  remains  considerable  need  to  develop  consensus  over  the 
factors  defining  quality  of  life,  the  appropriate  measurement  options,  tracking  and 
reporting  of  assessment  results,  and  the  utilization  of  results  and  analyses  in  directing 
policy  decisions  for  services  and  supports  designed  to  assist  people  with  developmental 
disabilities. 
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IV.  BRIEF  SUMMARY  OF  FINDINGS  AND  RECOMMENDATIONS 


Summary 

The  results  of  this  project  strongly  indicate  that  many  states  have  established  or  are  in 
the  process  of  developing  outcome-focused  quality  assurance  mechanisms  that 
encompass  the  selected  outcomes  as  well  as  other  important  benefits.  While  not  all 
states  have  developed  systematic  outcome  monitoring  approaches,  the  7  states  visited 
by  project  staff  are  committed  to  outcome  monitoring  and  are  variously  in  the  process 
of  imbedding  such  schemes  in  their  ongoing  quality  assurance/quality  enhancement 
systems.  On  the  basis  of  key  informant  interviews  in  the  states,  it  also  appears  that  the 
move  to  outcomes  was  accompanied  by  the  maintenance  of  health  and  safety 
protections  albeit  in  a  somewhat  different  form. 

Further,  the  analysis  of  the  psychometric  properties  of  the  instruments  used  in  each  of 
these  states  shows  that  the  majority  are  reliable,  applicable  to  institutional  populations 
and  discriminate  among  categories  of  consumers  in  an  acceptable  fashion.  Concurrent 
validity  was  found  in  three  of  the  most  established  state  QA  instruments  —  Colorado, 
Oregon,  and  Oklahoma,  but  was  inconclusive  or  inadequate  in  the  remaining  states. 
Finally,  no  significant  correlations  could  be  found  between  outcome  scores  and  health 
and  safety  indicators. 

Several  variables  were  important  in  the  success  of  the  systems  reviewed  in  the  study 
states  including  a  commitment  from  state  leadership,  broad  stakeholder  participation, 
the  provision  of  assistance  to  enhance  the  acceptance  of  outcome  monitoring, 
participation  of  family  members  and  people  with  disabilities,  commitment  of  direct 
support  staff,  and  development  of  a  continuous  quality  improvement  philosophy. 
Other  factors  also  appeared  to  play  an  important  role  as  catalysts  for  the  creation  of 
outcome  approaches  including  the  presence  of  a  major  lawsuit  and  the  infusion  of  new 
service  dollars  into  the  system. 

With  respect  to  the  identification  of  quality  indicators,  project  staff  listed  a  range  of 
areas  that  could  be  used  by  HCFA  as  a  basis  for  developing  quality  indicators  to  both 
monitor  the  well-being  of  people  living  in  ICFs/MR  as  well  as  their  progress  on  valued 
outcomes. 

In  addition  to  recommendations  included  in  each  project  report,  presented  below  are 
our  suggestions  for  concepts  that  HCFA  could  consider  as  they  build  the  future  quality 
assurance  system  for  this  program. 
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Performance  Contract 

HCFA  should  consider  further  investigation  into  the  value  of  entering  into  performance 
contracts  with  each  state  that  supports  people  in  ICFs/MR.  Any  such  approach  should 
include  the  following: 

•  Evidence  that  the  contract  represents  a  collaboration  between  the  Mental 
Retardation/Developmental  Disabilities,  the  state  certification  agency,  and  the 
state  Medicaid  agency. 

•  A  plan  that  includes  a  process  for:  1)  ensuring  that  ICF/MR  providers 
implement  a  continuous  quality  improvement  system,  2)  a  mechanism  for 
collecting  data  on  valued  outcomes,  3)  an  external  monitoring  process  that 
includes  citizens  and  consumers,  and  4)  a  process  for  building  organizational 
capacity  regarding  the  implementation  of  an  outcome-focused  approach  to  the 
provision  of  care  and  supports,  including  the  retraining  of  staff,  changes  in  staff 
performance  reviews,  and  proposals  for  monitoring  outcomes  within  an 
agency. 

•  A  proposal  to  monitor  the  health  and  safety  of  residents  of  ICFs/MR  including 
the  development  of  individual  health  and  safety  plans  where  appropriate.  Such 
an  agreement  should  also  include  an  affirmation  to  submit  data  on  selected 
health  and  safety  indicators  developed  by  HCFA. 

•  A  mechanism  to  ensure  that  residents  of  ICFs/MR  receive  active  treatment  (e.g., 
traditional  surveys,  accreditation,  or  other  certification  process). 

•  An  agreement  to  prepare  yearly  reports  regarding  that  status  of  performance 
contract  activities  and  the  steps  taken  to  enhance  care  and  supports  in 
ICFs/MR.  This  report  should  also  include  information  on  ways  in  which  the 
results  of  state  quality  assurance  efforts  are  made  available  to  consumers  of 
service  as  well  as  to  family  members  and  other  interested  parties. 

Performance  Indicators 

We  suggest  that  HCFA  develop  a  set  of  quality  of  life  and  health  and  safety  indicators 
based  on  the  methodology  described  in  the  report  submitted  pursuant  to  Task  B  of  this 
project.  The  development  of  such  a  data  set  would  require  an  exploration  of  those 
individual  indicators  currently  extant  that  are  most  likely  to  predict  conditions  that  may 
jeopardize  either  the  health  and  safety  or  the  quality  of  life  of  ICF/MR  residents.  These 
indicators  in  turn  should  be  used  as  "red  flags"  to  trigger  more  in-depth  review  by 
HCFA  regional  surveyors.  These  in-depth  "red  flag"  reviews  should  be  conducted 
using  current  federal  health  and  safety  standards.  We  further  suggest  that  HCFA  take 
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the  lead  in  developing  and  publishing  national  norms  for  these  indicators,  through 
periodic  national  surveys. 

Active  Treatment 

Our  opinion  in  reviewing  the  HCFA  ICF/MR  regulations  is  that  the  current  regulations, 
while  providing  significant  assurances  that  residents  of  ICFs/MR  receive  a  wide  range 
of  needed  interventions,  rely  too  heavily  on  professional  perspectives  and  are  perceived 
as  too  prescriptive. 

Although  the  most  recent  proposed  changes  in  the  active  treatment  regulations  (March 
18,  1996  memo)  move  in  a  positive  direction,  there  are  still  areas  that  can  be  improved 
including  a  de-emphasis  on  the  assessment  of  needs  in  favor  of  an  increased  emphasis 
on  eliciting  goals  and  preferences,  a  more  explicit  bridge  between  the  notions  of  active 
treatment  and  the  new  supports  orientation  of  the  AAMR  Definition,  and  a  more 
prominent  role  for  person-centered  planning.  Further,  it  is  suggested  that  HCFA 
consider  becoming  a  center  for  excellence  in  quality  assurance  with  responsibility  to 
distribute  state-of-the  art  information  on  innovative  quality  processes  and 
dissemination  of  analyses  of  quality  trends  based  on  the  collection  of  performance 
indicator  data. 

We  suggest  that  HCFA  continue  to  refine  the  definition  of  active  treatment  based  on  the 
recommendations  included  in  Report  #2.  We  further  suggest  that  surveyor  training 
curricula  be  supplemented  to  include  an  introduction  to  the  supports  model,  an 
orientation  to  the  new  definition,  a  discussion  of  person-centered  planning,  and  an 
examination  of  the  ways  in  which  to  elicit  responses  from  people  with  severe 
disabilities. 

Applicability  to  All  HCFA  Funded  Services  to  People  with  Mental  Retardation 

Several  factors  suggest  that  some  of  the  recommendations  listed  should  pertain  to 
persons  with  mental  retardation  regardless  of  where  they  live,  including  the  fact  that: 
1)  many  states  are  rapidly  downsizing  and  phasing  out  many  public  and  private 
ICF/MR  beds;  2)  managed  care  organizations  will  play  a  larger  role  in  the  allocation  of 
Title  XIX  funds  in  long-term  care;  3)  provider  networks  may  replace  traditional  sub- 
state  administrative  structures;  and  4)  many  people  who  previously  resided  in 
ICFs/MR  are  now  living  in  waiver  funded  programs. 

For  these  reasons,  we  would  strongly  suggest  that  at  least  those  portions  of  the 
recommendations  discussed  above  that  deal  with  quality  of  life  indicators,  continuous 
quality  improvement,  external  monitoring,  and  the  publication  of  quality  assurance 
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findings  be  included  in  requirements  for  the  Home  and  Community  Based  Waiver 
Programs  (1950(b))  as  well  as  included  in  the  criteria  for  the  approval  of  1115  managed 
care  waivers. 


64 


Report  #2 

Analysis  of  the  Application  of  the  Active  Treatment  Definition 

(Objective  5,  Task  A) 

REVIEW  OF  THE  CURRENT  DEFINITION  OF  ACTIVE 
TREATMENT  AND  RELATED  ICF/MR  QUALITY 
ASSURANCE  ACTIVITIES 


I 
I 
I 

I 


REPORT  #2 

ANALYSIS  OF  THE  APPLICATION  OF  THE  ACTIVE  TREATMENT 

DEFINITION 
TABLE  OF  CONTENTS 


BACKGROUND  1 

METHODOLOGY  3 

RESULTS  9 

Perspectives  on  Active  Treatment  10 

Active  Treatment  and  Person  Centered  Planning  12 

"Continuous  Active  Treatment"  and  "Over-programming"  13 

Active  Treatment  and  Making  Choices  14 

Active  Treatment  Compliance  and  Outcome-Based  Accreditation  ,  16 

Survey  Procedures  19 

Active  Treatment  and  Utilization  Review  20 

DISCUSSION  21 

IMPLICATIONS  AND  RECOMMENDATIONS  23 

Suggestions  for  Near-Term  Activities  23 

Activities  for  the  Longer  Term  27 

An  Outline  of  New  Support  Planning  Format  31 


I 
I 


REFERENCES  36 

Appendix  A  Knowledgeable  Individuals  on  ICF/MR  Quality  Assurance  who 

Provided  Information  in  Development  of  Task  A  Interim  Report 

Appendix  B  1        Behavioral  Development  Survey 

2        Health  and  Safety  Issues  Instrument 

Appendix  C  1        State  by  State  Analyses  of  Psychometric  Properties  of 

Outcome-Based  Quality  Assurance  Instruments 

2  Analysis  of  Accreditation  Council  Survey 

3  Analysis  of  Commission  on  the  Accreditation  of  Rehabilitation 
Facilities  Surveys 

Appendix  D  1        Comments  to  HCFA  on  Draft  Revisions  to  the  Survey  Process 

for  ICF/MR 

2        Comments  on  Revised  Conditions  of  Participation  for  ICFs/MR 
Appendix  E  Cumulative  List  of  all  Interviewees 

Appendix  F  Combined  References  (Tasks  A  &  B) 


ii 


I 
I 
I 
I 
I 


Report  #2 


Analysis  of  the  Application  of  the  Active  Treatment  Definition 

(Objective  5,  Task  A) 

c>  c>  c=>  d>  c>  c>  cj>  c>  c>  c>  c>  c>  .=>  c>  c>  .=> 

REVIEW  OF  THE  CURRENT  DEFINITION  OF  ACTIVE 
TREATMENT  AND  RELATED  ICF/MR  QUALITY 
ASSURANCE  ACTIVITIES 


BACKGROUND 

The  Health  Care  Financing  Administration  (HCFA)  has  asked  the  Human  Services 
Research  Institute  (HSRI)  to  make  determinations  about  those  aspects  of  HCFA's 
regulatory  definition  of  active  treatment,  including  HCFA's  interpretative  guidelines, 
that  are  consistent  with  contemporary  notions  of  support  and  those  that  hamper  such 
concepts.  Because  of  the  linkage  of  the  active  treatment  definition  and  guidelines  with 
other  aspects  of  ICF/MR  quality  assurance  (e.g.,  the  pilots  of  new  survey  approaches 
conducted  spring  1995  and  the  Independent  Professional  Review/Utilization  Review 
component  of  ICF/MR  quality  assurance),  HSRI's  review  encompassed  these  and  other 
related  elements.  Although  HCFA  requested  that  we  provide,  to  the  maximum  extent 
possible,  empirical  data  to  support  or  refute  our  determinations  (i.e.,  data  from  studies 
which  employed  statistically  valid  sample  sizes  and  methods),  no  such  data  were 
identified. 

The  definition  of  active  treatment  is  central  to  the  ICF/MR  quality  assurance  program 
in  at  least  four  ways: 
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1.  Condition  of  Participation  in  the  ICF/MR  program:  Facilities  must  meet  the  active 
treatment  Condition  of  Participation  (CoP)  at  6483.440  in  order  to  be  certified  as 
ICFs/MR. 

2.  Principal  focus  of  the  ICF/MR  survey:  Assessment  of  the  delivery  of  active 
treatment  services  is  identified  in  Appendix  J:  Survey  Procedures  and  Interpretive 
Guidelines  for  Intermediate  Care  Facilities  for  the  Mentally  Retarded  (HCFA,  1988; 
1996)  as  the  primary  focus  of  the  survey. 

3.  Determination  of  eligible  beneficiaries:  Only  individuals  with  mental  retardation 
or  related  conditions  who  are  in  need  of  continuous  active  treatment  are 
eligible  for  ICF/MR  services.  ICFs/MR  are  required  to  meet  the  standard  that 
"Clients  who  are  admitted  by  the  facility  must  be  in  need  of  and  receiving 
active  treatment  services"  (5483.440(b)(1)). 

4.  Reimbursement  allowances:  Facilities  may  receive  reimbursement  only  for  the 
cost  of  care  of  individuals  classified  for  the  ICF/MR  level  of  care  who  are 
receiving  active  treatment. 

The  active  treatment  CoP  section  of  Appendix  }  begins  with  the  active  treatment 
standard  at  W195,  followed  by  five  standards  and  70  areas  of  review  (tag  numbers): 

•  W196-197:  Active  treatment  definition; 

•  Wl 98-205:  Admissions,  transfers  and  discharge; 

•  W206-248:  Individual  program  plan,  including  IPP  development  by  the 
interdisciplinary  team,  the  comprehensive  functional  assessment; 

•  W249-251:  Program  implementation;  and 

•  W252-265:  Program  documentation,  including  program  monitoring  and 
change  as  well  as  requirements  for  a  specially  constituted  committee  and 
informed  consent  for  behavioral  training  programs  that  involve  risks  to  client 
protection  and  rights. 

Revisions  to  Appendix  },  Interpretive  Guidelines  for  ICFs/MR  effective  December  4,  1995, 
include  Condition  Level  Compliance  Principles  for  W195  and  Facility  Practice 
Statements  on  50  of  the  70  active  treatment  tag  numbers.  In  addition,  the  new 
Fundamental  Survey  described  in  the  revised  Appendix  J  to  be  effective  July  1, 1996,  and 
implemented  no  later  than  October  1, 1996,  calls  for  initial  review  of  a  core  group  of  15 
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tag  numbers  in  determining  the  Active  Treatment  CoP,  with  remaining  requirements 
considered  as  supporting  structures  or  processes  that  the  facility  must  implement.  The 
revised  Appendix  J  also  increases  the  survey's  focus  on  outcomes  and  on  observations 
and  interviews  as  the  primary  methods  of  information  gathering.  Surveyor  training 
taking  place  in  the  Spring  and  Summer  1996  incorporates  these  concepts  and 
approaches. 

HSRI's  review  of  the  active  treatment  definition  and  related  ICF/MR  quality  assurance 
activities  has  spanned  the  period  of  time  when  revisions  in  Appendix  J  were  being 
made.  Most  of  the  information  on  active  treatment  quality  assurance  analyzed  by  HSRI 
was  related  to  standards  and  surveys  based  on  the  1988  edition  of  Appendix  J.  Where 
appropriate,  however,  we  have  included  actual  and  proposed  revisions  to  Appendix  J  in 
our  review,  as  well  as  the  informal  draft  proposal  for  revised  ICF/MR  regulations 
(HCFA,  March  1996).  It  is  clear  from  many  of  the  actions  taken  by  HCFA  since  we 
began  our  review  in  late  1994  that  HCFA  is  moving  to  bring  active  treatment  quality 
assurance  more  in  line  with  contemporary  notions  of  state  of  the  art  service  delivery 
and  with  effective  outcome-based  quality  assurance  principles  that  are  compatible 
with  the  support  model. 

METHODOLOGY 

HSRI  used  six  sources  of  information  in  conducting  the  review: 

1.  Informal  discussions  with  State  contacts  across  the  nation  regarding  their 
experience  with  ICF/MR  quality  assurance  and  perspectives  on  the  active 
treatment  regulations  and  guidelines,  in  conjunction  with  the  development  of 
the  Task  A  Interim  Report.  A  total  of  60  informants  in  41  States  with  experience 
with  ICF/MR  surveys  provided  information  on  active  treatment  and  related 
ICF/MR  quality  assurance,  and  on  suggested  reforms  (names  and  titles  of  these 
individuals  are  provided  in  Appendix  A): 

•  Thirty-five  quality  assurance  staff  in  State  mental  retardation /developmental 
disabilities  agencies,  including  23  State  quality  assurance  directors,  6  quality 
assurance  staff  for  community  residential  services  or  other  State  quality 
assurance  responsibilities,  and  6  individuals  with  responsibility  for  quality 
assurance  in  one  or  more  State  facilities; 

•  Seven  directors  of  State  mental  retardation /developmental  disabilities  service 
agencies; 
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•  Sixteen  State  mental  retardation/developmental  disabilities  staff  with 
responsibilities  such  as  community  services,  policy  development,  and 
regional  service  administration; 

•  One  administrator  of  Medicaid  services  for  people  with  mental 
retardation /developmental  disabilities;  and 

•  One  provider. 

2.  Interviews  with  knowledgeable  individuals  during  site  visits  to  eight  States 
conducted  to  review  outcome-focused  quality  assurance  systems  (additional 
Task  A  activities  reported  on  in  a  separate  document),  including  State  mental 
retardation/developmental  disabilities  agency  staff,  ICF/MR  surveyors, 
QMRPs,  administrators  of  State  institutions  certified  as  ICFs/MR,  and 
providers  of  community  ICF/MR  services.  Names  and  titles  of  these 
individuals  are  provided  in  Appendix  E  (indicated  by  an  *). 

3.  Literature  on  ICF/MR  quality  assurance: 

•  Formal  literature  (e.g.,  peer  reviewed  journals),  including  ICF/MR  program 
history  and  perspectives  on  ICF/MR  quality  assurance;  and 

•  Informal  literature  (e.g.,  conference  presentations). 

As  noted  above,  the  literature  search  did  not  identify  any  empirical  studies  on 
consistency  between  ICF/MR  active  treatment  quality  assurance  and 
contemporary  notions  of  support 

4.  Input  from  members  of  the  Technical  Working  Group  and  their  respective 
organizations,  as  appropriate.  In  this  context,  project  staff  participated  in 
discussions  of  the  National  Association  of  Superintendents  of  Public 
Residential  Facilities  for  the  Mentally  Retarded,  the  American  Network  of 
Community  Residential  Options,  and  the  ICF/MR  Task  Force  of  the  American 
Health  Care  Association  regarding  ICF/MR  quality  assurance. 

5.  Telephone  interviews  with  additional  knowledgeable  informants  in  seven 
study  States,  targeted  specifically  to  members  of  ICF/MR  survey  teams, 
administrators  in  State  facilities  certified  as  ICFs/MR,  and  five  additional 
community  ICF/MR  providers  in  another  four  states.  As  approved  by  the 
HCFA  project  officer,  these  interviews  were  conducted  to  obtain  additional 
information  on  active  treatment  quality  assurance  in  lieu  of  HSRI's  direct 
observation  of  an  ICF/MR  survey.  Names  and  titles  of  these  individuals  are 
provided  in  Appendix  E. (indicated  by  an  *). 
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6.  HCFA  documents  on  ICF/MR  quality  assurance,  including: 

•  State  Operations  Manual:  Provider  Certification.   Transmittal  No.  212  (October 
1988). 

•  Draft  No.  6  Revised  Survey  Procedures,  July  1995. 

•  Revised  Exhibit  80  and  Appendix  }  (J-13  -  J-138),  December  1995. 

•  ICF/MR  Conditions  of  Participation  (Proposed  1996  Draft),  March  18, 1996. 

•  Draft  Revised  Exhibit  80,  Sections  2900-2901  and  Appendix  J  (J-l  -  J-20),  spring 
1996  (to  be  effective  July  1, 1996). 

Comments  on  the  Draft  No.  6  Revised  Survey  Procedures  and  on  the  informal  draft 
of  proposed  ICF/MR  Conditions  of  Participation  previously  submitted  are 
included  in  Appendix  D  of  this  report 

HSRI's  review  and  analysis  was  affected  by  three  factors:  the  changes  in  ICF/MR 
quality  assurance  being  undertaken  by  HCFA,  as  discussed  above;  the  need  to  broaden 
the  scope  of  our  analysis  beyond  the  definition  of  active  treatment  and  the  guidance  to 
surveying  the  active  treatment  CoP;  and  the  difficulty  in  determining  what  is  due 
solely  to  Federal  regulations  and  requirements  for  the  review  of  active  treatment 

Because  a  major  part  of  our  charge  was  to  look  at  active  treatment  in  relation  to 
contemporary  notions  of  quality,  HSRI  used  consistency  with  the  primary  outcomes 
defined  by  the  Technical  Working  Group  as  one  basis  for  its  analysis.  The  primary 
outcomes  are  as  follows: 

•  Self-determination  •  Personal  growth 

•  Dignity  •  Individualism 

•  Community  integration /inclusion  •  Health  and  safety 

•  Relationships/ social  connections  •  Consumer  satisfaction 

Both  "self-determination"  and  "individualism"  were  associated  with  the  concept  of 
consumer  choice,  an  identified  area  of  interest  to  HCFA  in  our  review  of  active 
treatment.  Perspectives  of  active  treatment  out  in  the  field,  however,  often  encompass 
standards  and  survey  issues  that  are  not  within  the  active  treatment  CoP,  such  as 
limitations  on  choice-making  in  the  dietetic  services  section  of  the  standards,  discussed 
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below.  In  addition,  some  of  the  guidance  that  is  particularly  strong  in  its  consistency 
with  choice  and  the  primary  outcomes  is  found  in  the  draft  revised  Appendix  J 
Compliance  Principles  and  Facility  Practice  statements  for  the  Client  Protections  CoP 
and  the  Client  Behavior  and  Facility  Practices  CoP. 

In  order  to  relate  the  monitoring  process  to  active  treatment  issues,  it  was  necessary  to 
review  the  survey  methodology  itself,  and  the  extent  to  which  the  review  of  active 
treatment  during  the  survey  process  (or,  in  some  cases,  the  anticipation  of  a  survey 
from  previous  experience)  affects  the  provision  of  state-of-the-art  services.  Specifically 
survey  procedures  were  reviewed  in  light  of  the  common  linkage  between  state-of-the- 
art  services,  in  particular  person  centered  planning  and  the  support  model,  and  the 
shift  to  outcome-focused  quality  assurance  methodologies.  This  linkage  was  apparent 
throughout  the  States  that  had  moved  to  more  outcome-based  quality  assurance,  as 
described  in  the  Interim  Task  A  Report. 

Regarding  perspectives  on  the  consistency  of  Federal  standards  with  contemporary 
concepts,  HSRI  found  throughout  its  analysis  that  it  is  virtually  impossible  to  isolate  the 
effects  of  the  Federal  regulations  and  survey  procedures  required  by  HCFA  in  isolation 
from  state  requirements.  In  some  cases,  State  and  local  regulations  that  go  beyond 
Federal  standards  are  a  source  of  problems  with  active  treatment  quality  assurance.  In 
a  California  facility,  for  example,  the  QMRP  reported  that  the  State  ICF/MR  survey 
team  cited  the  program  because  a  client's  Nintendo  game  was  stored  in  the  living  area 
with  the  facility's  entertainment  center  and  did  not  have  the  client's  name  on  it  nor  was 
it  stored  in  the  client's  room.  According  to  the  QMRP,  the  client  wanted  the  game  to  be 
available  to  his  roommates,  which  was  explained  to  the  surveyors.  The  citation, 
however,  was  for  W109~compliance  with  Federal,  State  and  local  laws  on  sanitation— 
rather  than  a  Federal  standard  requiring  labeling  or  storage  in  an  individual's  room. 
The  perception,  however,  was  that  it  was  ICF/MR  rules  that  were  interfering  with  a 
normalizing  activity. 

Similarly,  it  is  often  difficult  to  identify  misapplications  of  Federal  standards  by 
individual  survey  teams,  or  to  separate  perspectives  on  actual  regulations  from  those 
stemming  from  misapplications.  There  are  still  reports,  for  example,  of  surveyors 
requiring  specific  numbers  of  objectives  in  Individual  Program  Plans  (IPPs)  or  of  a 
predetermined  number  of  hours  of  day  services,  leisure  activities,  etc.,  in  order  to 
comply  with  active  treatment  standards.  Although  HCFA  continues  to  try  to  clarify 
that  these  kinds  of  requirements  are  not  part  of  the  Federal  regulations,  through  its 
guidelines  in  Appendix  J  and  surveyor  training  programs,  providers  continue  to  report 
that  some  surveyors  impose  them.  This  perpetuates  the  perception  that  it  is  ICF/MR 
standards  that  require  "over-programming"  or  excessive  paperwork.    Facilities  that 
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have  experienced  such  surveys  also  may  continue  to  respond  to  such  misapplications, 
in  fear  of  being  cited  again.  The  effects  of  misapplications  may  therefore  persist, 
despite  HCFA's  best  efforts  to  counteract  them. 

In  addition  to  the  primary  outcomes,  our  review  of  perspectives  on  the  active  treatment 
definition  and  its  use  in  ICF/MR  surveys  focused  on  concepts  of  the  support  model. 
The  evolution  that  has  taken  place  to  the  individual  support  model  is  summarized  in 
the  following: 
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TABLE  1 


FOCAL  QUESTIONS  FOR  THREE  HISTORICAL  ERAS 


Focal  Questions 

Era  of 
Institutions 

Era  of  Deinsti- 
tutionalization 

Era  of 

Community 
Membership 

Who  is  the  person  of 
concern? 

The  patient 

The  client 

The  citizen 

What  is  the  typical  setting? 

An  institution 

A  group  home, 
workshop, 
special  school,  or 
classroom 

A  person's 
home,  local 
ousiness,  tne 
neighborhood 
school 

How  are  the  services 
organized? 

In  facilities 

In  a  continuum  of 
options 

Through  a 
unique  array  of 
supports  tailored 
to  tne  muiviauai 

What  is  the  model? 

Custodial/medi 
cal 

Developmental/ 
behavioral 

Individual 
support 

What  are  the  services? 

Care 

Programs 

Supports 

How  are  services  planned? 

Through  a  plan 
of  care 

Through  an 
individualized 
habilitation  plan 

Through  a 
personal  futures 
plan 

Who  controls  the  planning 
decision? 

A  professional 
(usually  an  MD) 

An 

interdisciplinary 
team 

The  individual 

What  is  the  olannint? 
context? 

Standards  of 

professional 

practice 

Tpam  conspnsus 

A  cirri p  of 
support 

What  has  the  highest 
priority? 

Basic  needs 

Skill 

development, 

behavior 

management 

Self- 
determination 
and 

relationships 

What  is  the  objective? 

Control  or  cure 

To  change 
behavior 

To  change  the 
environment 
and  attitudes 

Adapted  from  "The  New  Service  Paradigm"  (Bradley,  1994). 
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RESULTS 


A  cross-section  of  individuals  (i.e.,  State  program  agency  officials,  surveyors  and 
ICF/MR  managers)  indicated  that  the  support  model  can  be  done  within  the  active 
treatment  regulations,  but  only  if  two  factors  are  present: 

1.  the  ICF/MR  is  firmly  committed  to  a  focus  on  outcomes  and  quality  of  life,  and 

2.  the  ICF/MR  surveyors  understand  the  value  of  the  support  model  and  can 
integrate  the  concept  into  an  active  treatment  framework. 

At  the  present  time,  however,  these  conditions  appear  to  be  the  exception  across  the 
country.  Despite  the  general  trend  toward  the  support  model  and  a  strong  focus  on 
consumer  centered  outcomes  and  quality  of  life,  there  is  considerable  variation  among 
ICF/MR  providers  in  the  degree  to  which  this  is  a  reality.  Similarly,  there  are  many 
differences  among  surveyors  in  their  understanding  of  contemporary  approaches  and 
in  their  commitment  and  comfort  level  with  an  outcome-based  survey.  Regulations 
may  therefore  be  applied  inconsistently  across  the  States.  Although  surveyor  training, 
coupled  with  HCFA  oversight  and  technical  assistance,  can  help,  it  is  likely  that 
differences  among  surveyors  will  persist  to  some  degree.  In  States  with  very  few 
ICFs/MR,  for  example,  or  those  with  no  small  community-based  ICFs/MR,  surveyors 
may  have  minimal  exposure  post-training  to  state  of  the  art  services,  especially  if  the 
majority  of  their  survey  work  is  done  in  nursing  facilities.  For  this  reason,  it  will  be 
necessary  for  HCFA  to  adopt  a  more  active  posture  in  assisting  surveyors  to  make  the 
connection  between  the  rhetoric  of  the  ICF/MR  standards  and  the  concrete  application 
of  the  support  model. 
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Perspectives  on  Active  Treatment 


The  term  "active  treatment"  has  been  central  to  the  ICF/MR  benefit  since  its  initial 
authorization  in  Public  Law  92-223  in  1972.  P.L.  92-223  reflected  Congressional 
recognition  that  the  needs  of  individuals  with  mental  retardation  often  differ 
significantly  from  elderly  long  term  care  beneficiaries  with  disabilities,  by  authorizing 
FFP  for  "rehabilitative  service"  as  well  as  for  health  care.  This  was  the  first  time  that 
Medicaid  FFP  was  authorized  for  institutional  care  that  was  not  primarily  medically 
oriented  or  only  "health  related"  (Boggs,  Lakin  &  Clauser,  1985).  These  principles  are 
addressed  in  the  regulatory  definition  of  active  treatment,  which  is  as  follows: 

Each  client  must  receive  a  continuous  active  treatment  program,  which 
includes  aggressive,  consistent  implementation  of  a  program  of  generic 
training,  treatment,  health  services  and  related  services  described  in  this 
subpart,  that  is  directed  toward  (i)  the  acquisition  of  the  behaviors 
necessary  for  the  client  to  function  with  as  much  self  determination  and 
independence  as  possible;  and  (ii)  the  prevention  or  deceleration  of 
regression  or  loss  of  current  optimal  functional  status  (42  CFR  483.440). 

HSRI  found  that  some  State  program  agency  staff  continue  to  see  active  treatment  as  a 
strength  of  the  ICF/MR  program,  as  opposed  to  custodial  care  and  lack  of  opportunity 
for  ICF/MR  residents  with  severe  disabilities.  These  officials  see  active  treatment 
requirements  as  a  way  to  prevent  a  return  to  custodial  care  because  of  the  assumption 
that  everyone  can  benefit  from  interventions— i.e.,  that  individuals  with  very  severe 
disabilities  are  not  "developmentally  incapable."  Additional  strengths  in  current 
ICF/MR  quality  assurance  noted  by  some  State  program  agency  informants  include: 

•  Help  from  Federal  on-site  training,  and  the  mentoring  role  played  by 
Federal  surveyors; 

The  importance  of  the  team  approach. 

•  Improvements  in  the  process  as  a  result  of  an  increased  emphasis  on 
observations  and  interviews. 

•  The  need  for  the  essential  minimum  protections  embodied  in  Federal 
standards. 

With  respect  to  constraints,  the  majority  of  State  program  agency  staff  and  ICF/MR 
managers  identified  one  or  more  of  the  following  concerns: 
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•  Compliance  does  not  equate  with  quality; 

•  The  standards  are  not  focused  on  outcomes; 

•  The  processes  that  are  reviewed  are  not  sufficiently  linked  to  desired  outcomes; 

•  The  yes/no  dichotomy  (i.e.,  in  or  out  of  compliance)  ignores  individual 
differences; 

•  The  adversarial  approach  of  the  survey  creates  an  atmosphere  of  fault  finding; 

•  There  is  no  of  focus  on  continuous  quality  improvement; 

•  The  survey  process  itself  is  intrusive  in  the  lives  of  people  with  mental 
retardation. 

These  observations  are  consistent  with  many  of  those  made  in  articles  on  ICF/MR 
regulation  such  as  Boggs  (1992),  Dillon  (1992),  Kennedy  (1990),  Holburn  (1992),  Shea 
(1992),  and  Taylor  (1992).  As  noted  above,  however,  no  empirical  studies  on  the  effects 
of  active  treatment  quality  assurance  were  identified. 

Similar  reports  of  constraints  on  contemporary  practice  were  are  found  in  comments  by 
members  of  the  Technical  Working  Group  and  related  national  organizations.  For 
example,  a  July  14, 1994  letter  from  the  American  Network  of  Community  Options  and 
Resources  (ANCOR),  an  organization  representing  service  providers,  including 
ICFs/MR,  to  Anthony  Tirone,  director  of  HCFA's  Office  of  Survey  and  Certification, 
Health  Standards  and  Quality  Bureau,  included  comments  on  the  need  to  redefine  the 
term  "active  treatment,"  as  well  as  concerns  about  an  earlier  (1994)  approach  to  Facility 
Practice  Statements  and  a  matrix  for  compliance  decision-making  (Savejic,  Johnson  & 
Fritz,  1994).  In  1995  an  ANCOR  task  force  on  ICFs/MR  indicated  that  surveyors 
continue  to  focus  on  process  and  paperwork,  there  are  difficulties  in  compliance  when 
using  person  centered  planning,  and  there  is  inconsistency  with  the  support  model 
(Gerowitz,  1995). 

Although  these  kinds  of  perspectives  may  be  based  on  the  imposition  of  State  and  local 
standards  and /or  on  surveyor  misapplication  of  the  Federal  standards,  they  contribute 
to  perceptions  that  the  ICF/MR  program  standards  are  inconsistent  with  contemporary 
concepts  and  approaches. 


11 


Active  Treatment  and  Person  Centered  Planning 


ICF/MR  surveyors,  providers  and  program  agency  staff  believe  that  the  ICF/MR  active 
treatment  regulations  can  (and  should)  be  made  to  accommodate  best  practice  efforts. 
At  its  origin,  active  treatment  was  designed  to  provide  stimulation  and  ongoing  activity 
for  consumers  who  otherwise  were  left  to  sit  alone  for  long  periods  of  time;  it  was  also 
geared  to  young  people  who  had  been  institutionalized  much  of  their  lives  and  needed 
significant  intervention  to  increase  their  life  skills.  Because  much  of  the  field  has 
moved  far  beyond  this  custodial  model,  and  because  the  ICF/MR  population  is  now 
substantially  older,  many  surveyors  and  ICF/MR  facility  managers  feel  that  the 
prescriptiveness  of  the  regulations  needs  to  be  reduced  and/ or  modified. 

One  of  the  major  issues  for  ICF/MR  facilities  moving  to  best-practice  approaches  is 
laying  out  in  advance  the  full  ramifications  of  the  new  practices  in  the  context  of  active 
treatment  -  in  short,  knowing  how  to  cross-walk  between  active  treatment  regulations 
and  the  new  approach.  Person-centered  planning  is  a  prime  example.  It  offers  much 
that  is  desirable  within  the  active  treatment  context  —  a  greater  role  for  consumers  and 
family  members  (W209),  attention  to  the  individual  increasing  his/her  self- 
determination  and  independence  (W196,  W240,  W247),  and  so  forth.  However,  other 
key  components  of  person-centered  planning  are  more  problematic  to  conceptualize 
within  the  active  treatment  framework,  such  as  the  strong  focus  on  consumer-preferred 
long-range  goals  and  the  development  of  natural  supports  in  the  community.  A  survey 
team  member  in  one  state  summed  up  the  problem  by  saying  that  "the  problem  is  that 
person  centered  planning  doesn't  cover  all  the  requirements  (the  tags)  that  are  part  of 
active  treatment." 

ICF/MR  surveyors  highlighted  a  key  difference  between  the  active  treatment  planning 
process  and  the  person-centered  planning  process:  the  relationship  between  needs  and 
goals  and  activities  on  the  other.  In  active  treatment,  the  starting  point  is  the 
individual's  needs,  as  identified  through  the  comprehensive  functional  assessment; 
these  needs  in  turn  should  be  tied  directly  to  the  goals  and  objectives  listed  in  the  IPP; 
finally,  these  objectives  should  lead  clearly  to  the  daily  activities  of  the  individual.  By 
contrast,  the  person-centered  planning  process  begins  with  identifying  the  individual's 
goals  and  objectives,  and  laying  out  activities  that  will  enhance  achievement  of  the 
goals.  In  applying  person-centered  planning  in  an  active  treatment  context,  the  goals 
must  be  translated  into  specific  needs  that,  in  turn,  are  addressed  through  particular 
objectives  and  thence  through  specific  activities.  Surveyors  expressed  support  for  the 
practice  of  beginning  with  goals,  but  struggled  to  find  evidence  that  any  subsequent 
link  was  made  to  needs  or  to  activities.  Their  experience  showed  that  person-centered 
planning  left  the  arena  of  activities  wide  open.  For  example,  what  do  you  do  when 
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staff  offer  consumers  choices  but  do  not  support  them  to  developing  choice-making 
skills  or  to  understand  the  consequences  of  their  choices.  What  should  you  do  when 
staff  support  consumers  to  participate  in  community  activities  but  do  not  assist  them  to 
develop  appropriate  behaviors  that  would  make  them  more  accepted  by  other 
community  members.  These  shortcomings  could  be  remedied,  as  the  surveyors  have 
suggested,  by  some  clear  thinking  about  needs  and  short-term  objectives  and 
measurement  of  progress.  The  situation  could  also  be  addressed  by  more  thorough 
training  and  mentoring  on  the  support  model,  in  particular  around  support/ training 
strategies  to  help  individuals  develop  needed  abilities. 

There  have  also  been  differences  in  the  experience  of  specific  ICFs/MR  that  have 
instituted  person  centered  planning  and  the  review  of  active  treatment  during  surveys. 
In  one  instance,  a  state  facility  in  Oregon  with  a  strong  person  centered  planning 
program  had  problems  meeting  active  treatment  requirements.  Surveyors  cited  a  lack 
of  formal  assessments  and  related  inability  to  measure  progress.  In  Utah,  however, 
surveyors  and  facility  administrators  indicated  that  person  centered  planning  fits 
within  active  treatment  regulations.  Similarly,  an  administrator/QMRP  at  a  State 
facility  in  Texas  reported  that  their  use  of  personal  futures  planning,  a  common  focus  of 
person  centered  planning,  has  not  been  a  problem  in  recent  surveys. 

"Continuous  Active  Treatment"  and  "Over-programming" 

One  of  the  recurring  criticisms  of  active  treatment  has  been  that  interpretations  during 
surveys  (or  in  anticipation  of  a  survey)  can  lead  to  over-programming.  This  concern 
was  heard  among  State  program  agency  officials  as  well  as  providers.  For  example, 
one  State  official  commented  that  when  active  treatment  is  expected  to  be  aggressive,  it 
is  sometimes  interpreted  as  requiring  constant  staff-client  interaction,  rather  than 
recognizing  that  clients  need  down  time.  Another  observed  that  "continuous"  was 
being  interpreted  as  requiring  programming  all  day  long  even  when  it  was 
meaningless  to  the  individual.  A  facility  administrator  in  Florida  provided  the 
following  example: 

During  a  recent  survey,  the  [survey]  team  required  that  K.S.  be  given  a 
formal  goal  related  to  her  production  at  the  day  program  where  she  is 
employed.  Vocational  skills  had  not  been  identified  as  a  barrier  for  her 
by  the  Interdisciplinary  Team.  In  fact,  K.S.  was  fully  integrated  into  the 
day  program  and  was  able  to  compete  any  task  which  was  assigned  to 
her.  She  continued  to  require  the  close  medical  supervision  provided  at 
the  program  and  therefore  was  not  at  that  time  a  good  candidate  for  a 
vocational  placement  in  the  community.  Communication  skills  had  been 
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identified  as  a  barrier  for  her,  and  she  was  making  significant  progress  in 
formal  programming  in  this  area.  The  team  would  not  accept  a 
communication  related  program  implemented  in  the  work  setting.  The 
result  was  that  K.S.  was  given  a  formal  goal,  related  to  her  production 
abilities  and  because  of  the  dynamic  nature  of  available  work  at  the  day 
program,  a  somewhat  artificial  practice  task  had  to  be  given  to  her  for  a 
program,  preventing  her  from  participating  naturally  in  the  day's 
activities,  as  did  her  peers  from  other  facilities.  This  issues  was  taken  to 
the  HCFA  Regional  Director  who  supported  the  surveyor  in  spite  of  all 
our  attempts  to  have  HCFA  understand  our  position.  In  the  end,  the 
citation  was  not  revoked. 

Although  this  reflects  a  misapplication  of  the  regulations,  the  reality  was  a  less 
desirable  program  for  K.S.  and  a  citation  for  the  ICF/MR. 

Another  widely  reported  concern  has  been  that  "continuous  active  treatment"  does  not 
allow  people  to  retire,  despite  guidance  in  Appendix  J  at  W196  and  W211  on  active 
treatment  for  elderly  individuals  clearly  indicating  an  age-appropriate  focus.  HSRI  did 
not,  however,  find  any  specific  examples  of  this  concern. 

Active  Treatment  and  Making  Choices 

State  program  agency  staff  and  providers  in  several  States  voiced  concerns  about 
restrictions  on  consumer  choice  and  decision-making.  The  project  team  contacted 
providers  in  approximately  six  States,  in  order  to  develop  more  specific  examples  of 
conflicts  between  the  regulations  and  choice.  Although  some  examples  illustrate 
problems  with  regulations  other  than  the  Federal  standards,  such  as  the  California 
Nintendo  game  issue  cited  above,  others  seemed  to  stem  from  Federal  ICF/MR 
standards.  The  examples  that  were  identified,  however,  were  not  necessarily  relevant 
to  the  standards  on  active  treatment.  For  example,  a  QMRP  in  a  Maine  facility  reported 
problems  regarding  choice  and  medication  reduction: 

There  have  been  instances  when  a  consumer  did  not  feel  comfortable  with 
medication  reduction,  but  a  decrease  was  clinically  indicated.  In  both 
cases  which  come  to  mind,  the  IDT  supported  their  decision  but  the 
facility  still  received  deficiencies.  Case  #1:  An  individual  with  a 
borderline  IQ  and  significant  mental  health  issues,  who  had  undergone 
reductions  (just  prior  to,  but  not  within  the  last  365  days)  and  was 
informed  about  her  medication  side  effects;  Case  #2:  A  person  receiving 


14 


Tegretol  for  seizure  activity  and  behavior  management  as  documented  by 
a  physician.  The  individual  clearly  expressed  concern  that  a  reduction 
could  result  in  seizure  activity.  People  appear  to  have  difficulty 
recognizing  the  difference  between  medication  used  to  "manage 
behaviors"  and  those  prescribed  to  address  the  symptoms  of  a  psychiatric 
disorder  (as  was  the  case  in  Case  #1).  There  have  been  cases  when  the 
IDT  reluctantly  recommended  a  medication  reduction  in  order  to  meet  the 
regulations  with  poor  results.  In  one  instance,  it  took  re-prescribing 
medication  (even  more  than  before),  and  months  for  the  person  to 
stabilize. 

Another  area  mentioned  by  several  provider  respondents  had  to  do  with  the  nutritional 
guidelines  regarding  the  timing  of  meals  and  the  resulting  constraints  on  choice 
regarding  the  schedule  and  location  of  eating  meals.  If  individuals  eat  breakfast  at  7:30 
a.m.,  then  they  must  eat  dinner  by  5:00  p.m.  —  not  usually  the  time  most  people  want  to 
go  out  to  dinner  and  enjoy  the  company  of  friends.  Further,  on  weekends,  when 
people  want  to  rise  late  and  eat  a  large  brunch  —  instead  of  breakfast  and  lunch,  this 
choice  is  precluded. 

Although  there  are  some  areas  of  the  regulations  that  directly  affect  choice  (e.g.,  the 
timing  of  meals),  most  of  the  providers  reported  that  the  abridgment  of  choice,  such  as 
the  examples  noted  above,  were  more  the  result  of  surveyor  interpretation.  One 
provider  from  Ohio  noted  that  after  working  with  surveyors  over  the  years,  they  came 
to  understand  the  larger  goals  of  integration  and  choice  and  became  more  flexible  in 
their  judgments.  There  is  also  support  for  consumer  choice  in  guidance  at  W247  on 
opportunities  for  client  choice  and  self-management  that  emphasizes  self- 
determination.  These  guidelines  note  that  "due  to  the  basic  underlying  importance 
'choice'  plays  in  the  quality  of  one's  life,  the  ICF/MR  should  maximize  daily  activities 
for  its  individuals  in  such  a  way  that  varying  degrees  of  decision-making  can  be 
practiced  as  skills  are  acquired."  Choice  also  is  referenced  in  guidelines  to  W250, 
stating  that  "individuals  should  have  opportunities  to  choose  activities  and  to  engage  in 
them  as  independently  and  freely  as  possible." 

Additional  support  for  an  emphasis  on  choice  is  found  in  the  Compliance  Principles  for 
W122,  the  CoP  for  client  protections.  References  to  choice  and  supports  for  increased 
choice  and  self-determination  are  weak,  however,  in  the  Compliance  Principles  for 
W195,  the  CoP  for  active  treatment.  Although  lack  of  opportunities  to  make  choices  in 
daily  routines  is  part  of  one  indicator  that  the  active  treatment  CoP  is  not  met,  there  is 
no  affirmative  principle  on  the  importance  of  individual  choice. 
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Active  Treatment  Compliance  and  Outcome-Based  Accreditation 

Another  area  of  inquiry  concerned  the  compatibility  of  ICP/MR  active  treatment 
quality  assurance  with  outcome-based  accreditation  systems.  Provider  respondents 
with  experience  with  both  approaches  believe  that  the  active  treatment  regulations  and 
the  Accreditation  Council  (AC)  outcome  standards  share  two  basic  characteristics:  both 
make  reference  to  the  contemporary  values  of  self-determination  and  choice,  age 
appropriateness  and  cultural  sensitivity,  and  community-based  activities;  and  both 
recognize  the  importance  of  training  and  teaching  to  improve  the  lives  of  people  with 
developmental  disabilities.  However,  beyond  this  commonality  in  orientation,  the  two 
processes  are  seen  to  differ  dramatically,  to  the  point  where  one  may  interfere  with  the 
successful  pursuit  of  the  other.  Fundamental  to  their  conflict  is  the  difference  in  approach 
to  quality.  The  active  treatment  regulations  are  perceived  as  focusing  on  forcing 
facilities  to  follow  certain  practices  and  requiring  certain  behavior,  while  the  premise  of 
the  AC  standards  is  that  organizations  must  facilitate  good  outcomes  by  allowing  a 
range  of  desired  practices  to  occur. 

This  same  contrast  ~  quality  control  as  opposed  to  quality  enhancement  —  carries 
through  to  the  ways  in  which  an  individual's  activities  are  structured.  Active  treatment 
requires  consistency,  careful  scheduling,  and  a  distinct  link  between  the  individual's 
daily  activities  and  the  objectives  in  his/her  Individual  Program  Plan  (IPP).  AC 
standards,  on  the  other  hand,  look  more  for  flexible,  individualized  activities  that  are 
tied  to  a  person's  preferences,  as  long  as  the  preferred  activity  contributes  to 
alleviating,  and  does  not  aggravate,  the  person's  problems  or  needs. 

In  similar  fashion,  the  fundamental  difference  in  approach  between  active  treatment 
regulations  and  AC  standards  results  in  contrasting  decision-making  processes.  Since 
active  treatment  means  particular  practices  must  occur,  and  that  each  activity  is  tied  to 
a  specific  objective  in  the  IPP,  professionals  are  needed  to  assure  that  things  occur  in  the 
prescribed  way.  By  contrast,  the  AC  standards,  in  allowing  for  flexibility  and 
differences  among  people,  foster  a  sense  of  openness  to  alternative  objectives  and 
activities,  and  look  first  to  the  individual  to  make  decisions,  as  well  as  to  non- 
professionals with  a  personal  connection  to  the  individual,  recognizing  that  these 
people  are  important  "quality  assurance  agents." 

Some  specific  examples  may  be  helpful  in  clarifying  the  way  that  active  treatment 
regulations  appear  to  interfere  with  the  ability  of  a  facility  to  attain  Accreditation 
Council  approval.  Included  here  are  insights  from  facilities  that  are  striving  toward  AC 
accreditation  and  one  which  has  been  AC  accredited. 
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The  professional  bias  in  active  treatment  requirements  is  most  apparent  in  the 
assessment  and  planning  tags.  The  requirements  for  the  Interdisciplinary  Team 
(IDT)  in  terms  of  composition  (W206-210)  and  responsibilities  (W21 1-239)  make 
assessment  and  planning  the  domain  of  the  professional.  Significant  attention 
is  given  to  establishing  a  linear  connection  between  an  individual's 
needs/deficits  as  identified  in  the  comprehensive  functional  assessment,  and 
the  IPP  objectives,  and  thence  to  specific  program  activities  (see  especially 
Survey  Procedures  IXC,  1988  version).  This  practice  puts  the  professional  in  the 
key  role  of  justifying  each  activity  that  the  individual  engages  in,  in  terms  of 
professionally-determined  needs  and  plan  objectives.  The  actual  planning 
meetings  may  well  include  very  active  individuals  and  family  members,  but 
professionals  take  responsibility  for  gathering  the  assessment  information 
(which  is  used  in  deciding  what  objectives  to  set)  and  in  arraying  the  possible 
program       activities       (which       will       address       the  objectives). 

A  few  facilities  have  explored  various  avenues  to  minimize  the  problem  of 
professional  dominance.  Two  of  the  more  successful  strategies  have  been  to 
decrease  the  role  of  specialists  in  service  provision,  and  to  increase  the 
interdisciplinary  focus  of  the  IDT.  As  direct  care  staff  take  on  more 
responsibility  for  physical  and  occupational  therapy  tasks  (although  these  are 
still  designed  by  the  PT  or  OT),  they  are  more  able  to  assert  the  consumer's 
wishes  and  strengths  into  the  planning  process;  this  works  best  where  staff 
have  a  long-standing  relationship  with  the  consumer.  Similarly,  IDT  members 
are  urged  to  rely  heavily  on  consumer  observation  and  interviewing  as  they 
develop  the  needed  assessments,  and  are  expected  to  present  their  own  "piece" 
in  the  context  of  the  whole  of  the  consumer's  life,  thus  giving  greater 
consideration  to  consumer  preferences  and  unique  personal  characteristics.  In 
essence,  these  facilities  have  influenced  the  professional  viewpoint  so  that  it 
reflects  more  of  the  consumer  perspective,  rather  than  trying  to  alter  the  power 
balance  directly. 

The  emphasis  on  having  and  following  a  detailed  activity  schedule  implies  a 
rigidity  in  activities,  despite  interpretive  guidelines  that  suggest  otherwise.  For 
example,  guidelines  for  W250  say  "the  schedule  provides  a  range  of  options, 
rather  than  a  fixed  regimen"  and  that  the  individual  should  experience 
"reasonable  choices."  However,  surveyors  are  only  directed  to  investigate 
further  when  they  observe  all  individuals  doing  the  same  activities  -  protecting 
against  the  problem  of  one  schedule  being  imposed  on  all  residents,  but  not 
highlighting  the  equal  problem  of  a  single  individual  being  unable  to  change 
his/her  schedule  on  short  notice,  which  is  a  more  normalized  pattern  of 
behavior.  Indeed,  the  1988  version  of  Appendix  J  that  was  in  use  in  the  field 
until  December  4,  1995,  when  a  revised  version  became  effective,  includes 
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probes  that  question  why  an  individual  is  not  following  the  prepared  schedule, 
specifically,  "what  is  the  individual  doing  when  not  'on  task'?  "for  how  long?" 

All  facilities  have  to  maintain  the  24-hour  schedule  for  consumers,  but  some  are 
more  willing  than  others  to  make  last-minute  changes  in  any  one  person's 
schedule.  The  key  seems  to  be  if  staff  feel  that  they  have  the  autonomy  to  make 
such  person-centered  changes,  and  that  they  will  be  supported  by  their 
supervisors,  should  any  questions  come  up.  Also  important  is  having  some 
flexibility  in  staff  assignment/availability  to  be  able  to  accommodate  each 
consumer's  wishes. 

•  Limitations  on  choice  and  individual  preferences  occur  throughout  the  active 
treatment  regulations,  but  are  perhaps  most  evident  for  the  dietetic  services 
condition  of  participation.  Tags  under  "meal  services"  include  a  requirement 
of  at  least  three  meals  daily  (W467),  at  regular  times  not  to  exceed  certain 
hourly  limits  (W468-471);  menu  requirements  prohibit  frequent  repetitions  of 
the  same  food  (W478-479),  although  the  probes  encourage  flexibility  as  long  as 
nutritional  value  is  comparable;  and  the  dining  areas  standard  requires 
individuals  to  eat  in  the  dining  area  unless  the  physician  or  the  IPP  documents 
the  necessity  of  doing  otherwise.  In  the  experience  of  State  facilities,  these  tags 
mean  that  the  individual  is  limited  in  when,  where,  and  what  he/she  eats,  even 
though  these  limitations  do  not  directly  increase  life  quality  and  may  even 
decrease  it.  Regulations  such  as  these  were  designed  to  remedy  gross  problems 
such  as  denying  people  food  or  not  giving  them  a  balanced  diet,  but  in  current 
practice  can  seriously  limit  the  goals  of  choice  and  individualization.  Other 
tags  foster  the  use  of  centralized  kitchens,  laundry,  and  pharmacy  services, 
because  of  the  necessity  to  ensure  control  of  specific  compliance  issues- 
Even  facilities  that  do  not  see  these  tag  limitations  as  significant  barriers  to 
consumer  choice  agree  that  they  can  be  problematic.  Again,  their  strategies 
stem  from  a  willingness  to  "stretch"  definitions  and  interpretations,  such  as 
designating  numerous  "dining  areas"  to  allow  choices  of  where  to  eat,  as  well 
as  a  willingness  to  complicate  the  responsibilities  of  supervisory  staff  (e.g.  the 
director  of  nutrition  services  sets  menu  selections  for  the  central  kitchen,  but 
also  oversees  the  food  shopping  list  and  menus  used  by  a  small  group  of 
residents  who  prepare  their  own  meals). 


Similarly,  State  facility  administrators  in  a  State  with  an  outcome-based  licensure 
system  modeled  after  the  AC  standards  reported  some  differences  between  the  State's 
quality  assurance  program  and  ICF/MR  quality  assurance: 
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Expectations  conflict  between  [the  State's  outcome-focused  quality 
assurance  system]  and  ICF/MR  quality  assurance.  The  ICF/MR  survey 
cited  wandering  in  the  hallway,  while  [the  outcome-based  survey]  said 
that  not  enough  people  were  wandering  in  the  hallways.  ICF/MR 
regulations  do  not  permit  [client]  rooms  to  be  locked,  yet  maybe  the 
individual  needs  privacy.  ICF/MR  rules  won't  permit  snaring  of  soap, 
but  [the  outcome-based  survey]  encourages  a  family  atmosphere.  Both 
systems  have  a  problem  with  imposing  value  systems.  For  example,  we 
can  be  alone  [if  we  choose],  but  these  systems  might  force  someone  to 
"integrate."  The  difference  is  that  ICF/MR  quality  assurance  would  force 
integration,  but  [the  outcome-based  quality  assurance  system]  would  look 
at  what  supports  you've  offered. 

The  major  factor  that  enables  ICF/MR  facilities  to  reduce  the  conflicts  between  active 
treatment  regulations  and  outcome  standards  is  the  meaningful  embrace  of  the 
outcome  philosophy.  Training  is  reported  to  be  crucial,  as  is  continued  mentoring 
among  staff  and  consistent  support  from  supervisors  and  administration  in  a  facility. 
Staff  longevity  is  also  important.  Where  staff,  especially  direct  care  staff,  have  been  in 
the  job  for  a  number  of  years,  and  have  established  meaningful  relationships  with 
consumers,  they  tend  to  be  more  open  to  explicitly  focusing  on  consumer  outcomes, 
and  may  have  already  been  "stretching7'  the  active  treatment  regulations  to  fit  an 
outcome  emphasis.  Ultimately,  however,  facility  staff  at  all  levels  have  to  know  that 
the  ICF/MR  surveyors  adhere  to  the  outcome  philosophy  and  will  be  open  to 
interpreting  the  regulations  in  a  way  that  is  most  meaningful  to  consumers.  Some 
surveyors  are  willing  to  discuss  the  particular  situations  with  facility  staff,  so  they 
understand  more  fully  what  is  occurring.  This  process  often  reduces  the  problem  to 
one  of  documentation,  which  can  be  more  readily  resolved  without  altering  the 
activities  for  the  consumer. 

Survey  Procedures 

The  focus  on  outcomes,  person  centered  planning  and  supports  to  reach  personal  goals 
lends  itself  more  to  the  type  of  survey  HCFA  has  developed  this  past  year  and  is 
currently  promoting  through  the  revised  survey  procedures  in  the  Appendix  J  that  will 
become  effective  July  1, 1996,  for  implementation  by  the  States  no  later  than  October  1, 
1996.  The  features  particularly  consistent  with  contemporary  concepts  include: 

•  The  priority  given  to  observations  and  interviews,  including  both  the 
individual  and  direct  care  staff; 
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•  The  use  of  a  Fundamental  Survey  scaled  back  to  focus  on  essential  rights  and 
services; 

•  Most  of  the  active  treatment  tags  selected  for  the  Fundamental  Survey. 

The  guidance  regarding  full  surveys  is  largely  unchanged  from  the  1988  version  except 
for  the  Compliance  Principles  and  Facility  Practice  Statements.  In  the  absence  of 
additional  guidance,  for  example  regarding  the  appropriateness  of  person  centered 
planning  in  IPP  development,  and  the  use  of  the  support  model  as  the  basis  for  "active 
treatment,"  traditional  interpretations  and  misapplications  of  the  ICF/MR  standards 
may  occur. 

Surveyors  also  may  require  more  support  on  the  effective  use  of  interviews,  especially 
in  interviewing  individuals  whose  verbal  ability  is  very  limited  or  non-existent.  As 
indicated  in  HSRI's  comments  on  Draft  No.  6  Revised  Survey  Procedures  (see  F),  this  is  an 
area  that  HCFA  may  want  to  strengthen. 

Active  Treatment  and  Utilization  Review 

During  the  course  of  its  investigation,  HSRI  received  scattered  reports  that  the  review 
of  active  treatment  in  conjunction  with  the  Independent  Professional 
Review /Utilization  Review  (IPR/UR)  process  is  more  inconsistent  with  contemporary 
concepts  than  the  ICF/MR  survey.  There  is  no  consensus  on  the  extent  of  the  problem, 
however,  due  to  the  wide  variation  among  the  States  on  how  IPR/UR  is  carried  out  in 
relation  to  the  ICF/MR  survey.  In  many  States,  IPR/UR  is  done  in  conjunction  with  the 
ICF/MR  survey,  by  the  ICF/MR  surveyors.  ICF/MR  providers  may  not  even  be  aware 
that  a  separate  process  has  occurred.  In  other  States,  however,  IPR/UR  is  done 
separately.  In  States  where  the  procedures  are  blurred,  providers  may  assume  that  the 
paperwork-focused  IPR/UR  review  is  part  of  the  ICF/MR  survey. 

Unlike  the  ICF/MR  standards  and  guidelines,  it  is  our  understanding  that  IPR/UR 
regulations  have  not  been  changed  since  1979.  They  seem  to  reflect  the  opposite  of 
HCFA's  current  ICF/MR  survey  focus  on  outcomes  and  primary  use  of  observations 
and  interviews.  Because  the  active  treatment  standards  require  that  ICFs/MR  provide 
continuous  active  treatment  (W196)  and  do  not  as  a  practice  serve  individuals  who  are 
not  in  need  of  continuous  active  treatment  (W197),  the  IPR/UR  review  also  is 
considered  duplicative  by  some  State  program  agency  administrators. 
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DISCUSSION 


Although  the  ICF/MR  active  treatment  regulations  can  be  made  to  accommodate  best 
practice,  and  to  allow  individuals  to  obtain  the  amount  and  variety  of  supports  they 
need  and  want,  this  does  not  mean  the  existing  regulations  should  not  be  changed. 
Ideally,  active  treatment  regulations  would  foster  positive  consumer  outcomes,  rather 
than  just  allow  them  to  occur.  Fundamentally,  the  prescriptiveness  of  the  current 
regulations  needs  to  be  reduced.  This  has  been  addressed  somewhat  through  the 
interpretive  guidelines  and  the  probes  offered  in  the  December  1995  version  of 
Appendix  /,  but  even  that  leaves  surveyors  and  program  managers  attending  to  minor 
details  at  the  expense  of  addressing  the  overall  quality  of  a  person's  life. 

There  is  strong  consensus  that  custodial  care  is  inappropriate,  and  that  some  degree  of 
regulation  is  necessary.  As  summarized  by  Taylor  (1992)  regarding  the  cross-section  of 
articles  cited  above,  none  of  the  contributors  advocated  doing  away  with  all 
regulations.  Similarly,  none  of  the  knowledgeable  individuals  across  the  States  whom 
we  spoke  with  and  no  members  of  the  Technical  Working  Group,  to  our  knowledge, 
advocate  total  deregulation  of  the  ICF/MR  program  regarding  the  provision  of  services 
and  supports.  It  is  also  understood  that  the  term  "active  treatment"  is  provided  in 
statute  governing  the  ICF/MR  program,  and  therefore  that  the  term  must  be  retained  in 
some  form  in  revised  regulations. 

HCFA's  actions  over  the  past  several  months,  as  well  as  actions  underway  as  this  report 
is  being  prepared  to  bring  updated  training  to  ICF/MR  surveyors  nationwide,  are 
clearly  steps  in  the  right  direction.  The  first  informal  draft  of  revised  ICF/MR 
standards  circulated  spring  1996  reflects  approaches  that  are  more  consistent  with 
contemporary  approaches  and  the  primary  outcomes.  Given  the  likelihood  that  it  will 
be  a  few  years  before  revised  ICF/MR  regulations  are  final,  however,  there  are  some 
steps  that  HCFA  can  take  to  make  current  standards,  guidance  and  survey  procedures 
more  consistent  in  the  interim. 

Overall,  it  appears  that  key  elements  consistent  with  the  primary  outcomes  and  with  a 
supports  model  are  present  in  current  regulations  and  guidance.  The  Compliance 
Principles  for  the  active  treatment  CoP  describe  outcomes  of  increased  skills  and 
independence  in  functional  life  areas,  one  of  which  is  use  of  community,  and  indicate 
that  the  CoP  is  not  met  when  individuals  are  not  involved  in  activities  which  address 
their  individualized  priority  needs  or  do  not  have  opportunities  to  practice  new  or 
existing  skills  and  to  make  choices  in  their  daily  routines.  Because  the  developmental 
approach  is  still  so  prominent,  however,  the  scattered  references  to  choice,  self- 
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determination,  community  involvement,  and  relationships  get  lost,  especially  in  the 
review  of  active  treatment.  In  particular,  despite  a  few  references  to  choice,  there  are 
no  explicit  references  to  active  treatment  based  on  the  goals  that  are  important  to  the 
individual. 

Further,  while  the  new  Facility  Practice  Statements  at  W213  on  identifying 
developmental  strengths  in  order  to  include  clear  descriptions  in  the  CFA  of  the 
individual's  preferences,  methods  of  coping/compensation,  friendships  and  positive 
attributes  is  a  step  forward,  it  is  offset  by  tag  W214.  That  tag  on  developmental  and 
behavioral  management  needs  calls  for  descriptions  of  individual  needs,  skill  deficits 
and  functional  limitations.  In  addition,  Facility  Practice  Statements  at  W227  (one  of  the 
15  tags  included  in  the  Fundamental  Survey  on  active  treatment)  require  that  the  IPP 
contain  a  list  of  specific  objectives  based  on  needs  identified  in  the  CFA,  a  clear  link 
between  the  specific  objectives  and  the  functional  assessment  data  and 
recommendations,  along  with  objectives  for  those  needs  that  are  observed  to  most 
likely  impact  on  the  individual's  ability  to  function  in  daily  life.  Guidelines  imply  a 
requirement  that  objectives  must  be  stated  for  the  needs  identified  in  each  domain 
included  in  the  CFA,  with  no  mention  of  objectives  selected  because  of  their  importance 
to  the  individual's  attainment  of  personal  goals. 

In  addition,  the  December  1995  revisions  to  Appendix  J  strengthen  the  active  treatment 
focus  on  the  acquisition  of  skills  in  activities  of  daily  living  (ADLs).  This  is  particularly 
noticeable  in  the  Facility  Practice  Statements  and  Guidelines  at  W242,  which  makes  the 
list  of  "personal  skills  essential  for  privacy  and  independence"  in  current  ICF/MR 
regulation  (s483.440(c)(6)(iii))  the  basis  for  required  training  programs  for  all 
individuals  who  lack  the  skills  listed,  and  describes  "the  receipt  of  training  targeted 
toward  amelioration  of  these  most  basic  skill  deficit  areas"  as  a  "critical  component  of 
the  active  treatment  program  needed  by  individuals  who  are  eligible  for  the  ICF/MR 
benefit"  that  therefore  "is  a  required  ICF/MR  service."  The  revised  Guidelines  further 
state  that  it  must  be  clear  "that  the  facility  has  organized  its  services  to  emphasize 
training  in  these  areas."  The  overall  impression  left  by  these  statements  is  that  active 
treatment  is  primarily  about  deficits  in  ADL  skills,  not  about  primary  outcomes  such  as 
community  integration  and  inclusion,  relationships,  and  self-determination.  This 
appears  inconsistent  with  the  introduction  to  survey  procedures  in  the  July  1996 
revisions  to  Appendix  J,  which  indicates  that  facilities  are  being  given  "broader  latitude 
to  develop  the  processes  by  which  it  implements  active  treatment  services." 

As  noted  above,  some  criticisms  of  the  standards  and  the  survey  process  appear  to  be 
the  result  of  State  and  local  regulations  that  go  beyond  Federal  requirements  or  of 
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misapplications  of  Federal  standards.  Training  and  dissemination  that  includes 
ICF/MR  managers  as  well  as  surveyors  may  reduce  these  misunderstandings. 

IMPLICATIONS  AND  RECOMMENDATIONS 
Suggestions  for  Near-Term  Activities 

There  are  implications  for  revised  definitions,  standards  and  survey  design  that  would 
help  bring  ICF/MR  quality  assurance  more  in  line  with  contemporary  values  and 
approaches  to  supporting  individuals  with  mental  retardation  /developmental 
disabilities.  In  the  short-term,  HCFA  should  consider  the  following  activities  to 
continue  its  move  in  that  direction: 

Training/Training  Support 

1.  Clarify  that  person-centered  planning  and  personal  futures  planning  can  be  the 
basis  for  the  IPP  and  for  the  active  treatment  being  provided  —  couched  in 
terms  of  the  CFA,  if  necessary  —  and  provide  written  guidance  on  how  to 
crosswalk  these  planning  approaches  with  the  current  regulations. 

2.  Include  training  on  person-centered  planning  itself,  using  resources  such  as 
Michael  Smull  of  the  University  of  Maryland  (recommended  because  of  his 
experience  with  large  State  facilities  as  well  as  community  residential  service 
clients). 

3.  Clarify  that  the  support  model  can  be  active  treatment,  coupled  with  revisions 
to  Facility  Practice  Statements  and  Guidelines  at  W242  to  indicate  that  skill 
building  and  behavioral  objectives  related  to  ADLs  are  a  component  of  active 
treatment,  but  not  the  only  one.  This  is  consistent  with  the  introduction  to 
survey  procedures  on  page  J-2  of  the  revision  effective  July  1,  1996,  as  well  as 
more  consistent  with  contemporary  practice. 

4.  Continue  the  training  on  observations  and  interviews  to  determine  active 
treatment  compliance,  with  expanded  coverage  on  the  use  of  observations  and 
interviews  with  and  in  relation  to  individuals  whose  communication  modes  are 
limited  or  primarily  non-verbal. 

5.  Include  examples  of  promoting  choice  and  providing  supports  to  self- 
determination,  relationships  and  community  participation  for  individuals  in 
large  institutional  ICFs/MR,  and  reinforce  surveyors'  expectations  that  these 
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primary  outcomes  are  not  just  for  individuals  in  smaller  community-based 
ICFs/MR. 

6.  Reinforce  the  clarification  on  retirement  as  an  appropriate  active  treatment 
option  for  older  individuals. 

7.  Review  the  training  video  about  to  be  distributed  to  be  sure  it  is  consistent  with 
these  approaches. 

8.  Work  with  Regional  Office  staff  to  identify  ongoing  needs  for  training  and 
technical  assistance,  especially  in  States  where  surveyors  review  a  relatively 
small  number  of  ICFs/MR. 

Revisions  and  Addenda  to  Appendix  J 

1.  Revise  the  Compliance  Principles  for  the  active  treatment  CoP  (W195)  to  move 
involvement  in  activities  to  meet  individualized  priority  needs  and 
opportunities  for  choice  to  the  principles  governing  when  the  CoP  is  met. 

2.  Rephrase  guidance  on  "continuous"  to  reinforce  the  emphasis  on  consistency 
and  coordination,  clarifying  that  this  does  not  mean  "every  waking  moment." 

3.  Revise  Facility  Practice  Statements  and  Guidelines  for  the  CFA  and  IPP  to 
include  person-centered  planning  as  a  desirable  approach  to  the  development 
of  active  treatment  programs.  Provide  a  specific  crosswalk  between  person 
centered  planning/personal  futures  planning  and  existing  regulations  on  the 
CFA  and  IPP. 

4.  Provide  guidance  on  expectations  for  ICFs/MR  in  understanding  the 
communications  of  people  with  severe  disabilities  and  understanding  and 
honoring  their  choices. 

5.  Include  in  the  Facility  Practice  Statements  questions  on  determining  people's 
preferred  lifestyles  and  in  providing  services  consistent  with  their  preferences. 

6.  Add  person-centered  planning  and  related  concepts  to  examples  of  in-service 
training  to  look  for  (W189)  and  demonstration  of  staff  skills  (W194). 

7.  Revise  the  list  of  suggested  interview  questions  and  guidance  on  interviewing 
individuals  to  incorporate  diverse  and  less  complex  communication  that  is 
more  appropriate  to  people  with  severe  cognitive  disabilities.  Emphasize  the 
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importance  of  talking  with  those  who  know  the  person  best,  including  direct 
care  staff  in  particular,  and  family  members /friend  advocates  as  appropriate, 
as  well  as  the  individual. 

8.  Consider  that  attendance  at  IPP  planning  sessions  is  not  the  only  way 
individuals  participate  in  IPP  development.  Although  surveyors  should 
question  how  the  individual's  goals,  wishes,  preferences  and  priorities  were 
central  to  IPP  development  and  review  of  progress,  physical  attendance  at  the 
meeting  is  not  always  the  best  method. 

9.  Modify  Facility  Practice  Statements  and  Guidelines  at  W242  as  indicated  above. 

10.  Cross-reference  relevant  standards  and  guidance  in  other  parts  of  the 
regulations,  in  particular  Compliance  Principles  for  W122,  the  CoP  on  Client 
Protections;  Facility  Practice  Statements  for  W125  on  teaching  and  encouraging 
individuals  to  claim  and  exercise  their  rights;  related  Facility  Practice  Statement 
at  W267  on  helping  individuals  to  learn  and  exercise  their  rights;  Compliance 
Principles  for  W266,  the  CoP  on  Behavior  and  Facility  Practices;  and  Facility 
Practice  Statements  at  W269  that  are  strongly  focused  on  the  key  role 
preferences  and  choices  should  play. 

11.  Include  W122,  W266  and  W269  in  the  Fundamental  Survey. 

12.  Cross-reference  recommended  revisions  on  the  CFA  and  the  IPP  to  W159  on 
QMRP  responsibilities  and  to  W164  re:  needs  for  professional  services. 

13.  Although  guidance  at  W136  includes  references  to  community  participation 
that  are  consistent  with  the  primary  outcome,  it  needs  to  be  stronger  on  choice. 
We  recommend  adding  guidelines  similar  to  those  at  W147,  which  include 
choice  in  relation  to  outings. 

14.  Examine  the  1992  AAMR  definition  of  mental  retardation  for  its  elucidation  of 
the  support  model  (i.e.,  not  necessarily  in  relation  to  defining  eligibility  for 
ICF/MR  services).  See,  for  example,  the  forthcoming  August  1996  issue  of 
Mental  Retardation,  which  will  include  guidelines  for  defining  levels  of  support. 

Research  and  Technical  Assistance  Support 

1.  Conduct  organized  "feedback  analysis"  of  surveyors  and  public  and  private 
ICF/MR  providers  from  October  1,  1996  -  September  30,  1997,  and  use  it  to 
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guide  training,  technical  assistance,  research  priorities,  interim  revisions  to 
Appendix  /,  and  revisions  to  ICF/MR  regulations. 

2.  Develop  HCFA's  ICF/MR  program  role  as  a  "Center  for  Excellence"  on  quality 
assurance. 

3.  Identify  resources  on  quality  assurance  appropriate  to  the  ICF/MR  program 
and  disseminate  information  on  them  to  State  ICF/MR  quality  assurance 
programs. 

4.  Incorporate  findings  from  the  Minnesota  quality  assurance  demonstration 
program,  the  Accreditation  Council  research  project,  and  others,  including  State 
data  collection  and  analysis  projects,  as  appropriate. 

5.  Develop  a  research  agenda  on  quality  assurance  methodologies  and  on 
outcomes  based  on  a  person  centered  planning/ individual  supports  model. 

6.  Examine  research  recently  conducted  by  Conroy  (1995)  comparing  outcomes 
for  matched  pairs  of  individuals  in  ICFs/MR  and  non-ICFs/MR,  considering 
potential  replication  and  implications  for  revisions  to  ICF/MR  quality 
assurance. 

7.  Continue  to  refine  and  disseminate  training  resources. 
Consensus  Building 

1.  Continue  plans  to  encourage  States  to  bring  providers  and  surveyors  together 
in  training  sessions  on  revised  survey  procedures. 

2.  Continue  wide  dissemination  of  further  refinements  of  the  proposed  revised 
standards. 

3.  Circulate  proposed  interim  revisions  to  Appendix  J  for  informal  review  and 
comment  among  stakeholders. 

4.  Promote  dissemination  of  the  Task  A  Final  Report,  this  report,  and  the  Task  B 
Final  Report  for  this  project. 

5.  Continue  plans  to  convene  focus  groups  of  stakeholders,  coupled  with  cross- 
cutting  groups  at  key  points  in  decision-making. 
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IPR/UR 


1.  Continue  efforts  to  eliminate  IPR/UR  regulations  as  redundant. 

2.  In  the  interim,  consider  issuing  guidance  to  the  States  cross-walking  IPR/UR 
with  relevant  portions  of  Appendix  J  with  revisions  as  suggested  above  (i.e., 
indicate  connection  to  survey  findings  at  W196  and  W242,  but  only  with 
revisions  to  clarify  that  active  treatment  is  more  than  building  ADL  skills. 

Activities  for  the  Longer  Term 

Revised  ICF/MR  Standards  and  Survey  Procedures 

HSRI  has  submitted  comments  on  the  informal  draft  revisions  of  March  1996  (see 
Appendix  D).  Our  assessment  is  that  they  reflect  a  positive  effort  to  consolidate 
standards,  reduce  confusion,  and  incorporate  the  primary  outcomes  —  at  least  up  to  a 
point.  The  format  is  consistent  with  our  suggestion  that  relevant  processes  that 
promote  valued  outcomes  be  included.  For  a  re-defined  and  re-oriented  active 
treatment  application,  this  suggests  that  surveyors  look  at  the  involvement  of 
individuals  and  those  closest  to  the  individuals  in  defining  what  is  important,  what  the 
consumers  hope  to  get  from  ICF/MR  services,  what  kinds  of  supports  are  most  helpful, 
and  what  is  affecting  the  move  toward  consumers'  personally  desired  outcomes, 
whether  positive  or  negative.  This  is  consistent  with  the  approach  taken  by  the 
Accreditation  Council  in  its  1992  standards,  which  includes  review  of  the  processes  as 
well  as  the  outcomes  they  relate  to. 

To  bring  the  revised  standards  even  more  in  line  with  contemporary  concepts,  we 
specifically  recommend  the  following  definition  of  active  treatment: 

Individualized  Services  and  Supports  (Active  Treatment)  Definition. 

(1)  Each  individual  must  receive  individualized  services  and  supports  (active 
treatment)  based  on  personal  goals,  preferences  and  needs.  The 
individualized  active  treatment  program  must  be  implemented  consistently 
and  include  services  and  supports  needed  by  each  individual  to: 

Develop  increasing  skills,  especially  those  that  promote  greater 
independence,  self-determination,  and  achievement  of  goals  important 
to  the  individual; 
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-  Develop  and  maintain  satisfying  relationships; 

-  Participate  in  community  life; 

-  Exercise  rights  and  responsibilities; 

-  Engage  in  productive  activities;  and 

-  Enjoy  and  participate  in  social,  leisure  and  recreational  activities. 

(2)  Active  treatment/individualized  services  and  supports  must  prevent  or 
decelerate  regression  or  loss  of  current  optimal  functional  status. 

(3)  Active  treatment  does  not  include  supports  or  services  to  maintain 
generally  independent  individuals  who  are  able  to  function  with  little 
supervision  or  in  the  absence  of  an  active  treatment  program  [same  as 
proposed  #2]. 

"Active"  can  be  defined  to  mean  the  provision  of  supports  and  services,  as  opposed  to 
custodial  care,  and  "treatment"  should  be  clarified  to  encompass  a  variety  of  supports 
and  services,  not  limited  to  professional  services  or  to  those  provided  by  and  in  the 
ICF/MR.  "Continuous"  could  be  redefined  to  mean  an  ongoing,  flexible  program  of 
individualized  services  and  supports,  emphasizing  that  this  does  not  mean  "every 
waking  moment"  interventions. 

HSRI  recommends  putting  the  following  as  the  first  active  treatment  outcome: 

Each  individual  receives  individualized  services  and  supports  (active 
treatment  program)  that  is  based  on  their  person-centered  plan,  in  which 
the  individual  and  those  who  know  him/her  best  have  identified 
personal  goals  and  the  services  and  supports  needed  to  reach  those  goals. 

We  also  recommend  that  proposed  outcome  #6,  on  individuals  identifying  and 
working  toward  the  accomplishment  of  personal  goals,  follows  this  outcome. 

Regarding  Active  Treatment  Facility  Processes,  we  recommend  that  the  first  process 
(revised  from  proposed  #4)  be  as  follows: 
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The  facility  implements  a  system  to  ensure  that  individual  personal  goals 
and  desires  are  identified  and  form  the  basis  of  his/her  plan  of  services 
and  supports. 

We  also  recommend  adding,  at  the  end  of  proposed  process  #7,  "and  to  reduce  the  need 
for  behavior  management  programs,"  as  well  as  adding  the  following  new  process: 

Supports  are  provided  to  help  individuals  develop  and  maintain 
relationships,  to  increase  their  participation  in  community  life,  and  to 
have  experiences  that  increase  their  ability  to  make  choices. 

We  recommend  revising  proposed  process  #10  as  follows: 

The  facility  ensures  that  individual  plans  are  modified  or  changed  in 
response  to  the  individual's  specific  accomplishments,  changes  in 
individual  goals  and  preferences,  need  for  new  programs,  or  difficulties 
in  acquiring  or  maintaining  skills. 

It  is  also  suggested  that  the  comprehensive  functional  assessment  and  IPP  be  replaced 
with  person-centered  planning/individual  support  plans,  without  prescribing  a 
particular  model.  Person-centered  planning  concepts  could  be  included  in  standards 
and  in  guidance  to  surveyors  on  what  to  look  for  in  personal  futures  planning  and 
other  similar  models  that  start  with  what's  important  to  the  individual  rather  than  what 
a  team  of  professionals  think  is  important.  For  example,  personal  futures  planning: 

•  Is  based  on  a  positive  view  of  people  with  disabilities,  rather  than  seeing  them  as 
having  "defects"  and  impairments  to  be  "fixed;" 

•  Is  comprised  of  goals  that  anticipate  that  positive  changes,  activities  and  experiences 
will  increase,  rather  than  focusing  on  decreasing  specific  negative  behaviors; 

•  Uses  ideas  and  possibilities  for  community  sites  and  settings,  and  valued  roles  in 
those  settings;  and 
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•  Includes  goals  that  are  important  to  the  individual  even  if  some  of  them  seem 
outlandish,  unrealistic,  and  impractical  or  require  major  changes  in  existing  patterns 
(adapted  from  Mount,  1994). 


An  example  of  a  generic  approach  to  such  planning,  developed  by  Michael  Smull  and 
distributed  at  the  Technical  Working  Group  meeting  on  December  11-12,  1995,  is 
presented  on  the  following  pages. 
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An  Outline  of  New  Support  Planning  Format 


A  new  process  can  be  seen  as  a  series  of  questions  and  answers,  assessment  and 
outcomes.  For  people  with  extensive  support  needs  an  outline  of  the  new  process 
would  be: 


TABLE  2 

PERSON  CENTERED  PLANNING 


Assessment 

Learn  how  the  person  wants  to  live 

Outcome 

Prioritized  description  of  how  the  person  wants  to  live 

Assessment 

Determine  if  there  are  any  issues  of  health  or  safety  in  the  context  of 
how  the  person  wants  to  live 

Outcome 

Description  of  how  issues  of  health  and  safety  will  be  addressed  in  the 
context  of  how  the  person  wants  to  live 

Assessment 

Compare  how  the  person  lives  now  with  how  they  want  to  live 

Outcome 

For  those  aspects  of  the  person's  life  that  are  congruent  with  how  they 
want  to  live  -  a  description  of  how  those  aspects  will  be  maintained 

For  those  aspects  that  are  not  congruent  with  how  they  want  to  live  -  a 
description  of  what  can  be  changed  in  the  short  term  (and  how  it  will  be 
changed)  and  what  will  take  time  (and  what  will  be  done).  Financial 
limitations /cost  barriers  will  be  identified  and  addressed  here. 

Assessment 

Within  the  context  of  maintaining  those  things  that  are  congruent  and 
changing  those  that  do  not  - 

vviiai  UUcs  liic  UclsUll  Wdili  IU  icdill  aim  IIUW  Lall  11  Uc  UaUglU; 

What  do  the  people  supporting  think  that  the  person  needs  to  learn,  is 
the  person  interested  in  learning  it,  and  how  can  it  be  taught? 

What  support  and/ or  assistance  does  the  person  need  and  how  will  it 
be  provided? 

How  will  the  people  implementing  the  plan  know  if  it  is  "working", 
what  are  the  verbal  and/or  behavioral  indicators  that  change  is  needed? 

Outcome 

An  individual  support  plan  that: 

•.   describes  what  will  be  done  and  who  will  do  it 

is  written  in  language  that  the  people  implementing  understand 
documents  that  the  outcomes  are  occurring;  and 

•.   documents  how  well  the  plan  is  working 
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There  should  be  a  relationship  between  the  amount  of  support  that  a  person  wants  and 
requires  and  the  length  and  complexity  of  the  plan.  For  people  with  few  support  needs 
the  planning  process  should  be  simpler.  For  those  people  who  are  quite  clear  about 
what  they  want  and  do  not  want  from  the  system  it  might  consist  of  a  process  as 
follows: 


TABLE  3 


PROBES:  PERSON-CENTERED  PLANNING 


Assessment 

Ask  the  person  what  they  want  from  the  service  system 

Outcome 

A  description  of  the  services  that  the  person  wants 

Assessment 

Determine  if  there  are  any  issues  of  health  or  safety  in 
the  context  of  how  the  person  wants  to  live 

Outcome 

Description  of  how  issues  of  health  and  safety  will  be 
addressed  in  the  context  of  how  the  person  wants  to 
live 

Assessment 

How  can  the  desired  supports  be  best  provided  given  - 

How  the  person  wants  to  live  and 

The  community  and  service  system  resources 

Outcome 

An  individual  support  plan  that: 

•.  describes  what  will  be  done  and  who  will  do  it 

•.  is  written  in  language  that  the  people  implementing 
understand 

•.  documents  that  the  outcomes  are  occurring;  and 
•.  documents  how  well  the  plan  is  working. 

Source:  M.  Smull  (1995). 


Survey  Procedures 

The  need  for  new  approaches  suggests  that  HCFA  is  on  the  right  track  in  emphasizing 
observation  and  interviews  over  record  review  in  its  revised  survey  protocols  piloted 
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spring  1995.  Two  specific  suggestions,  also  provided  in  HSRI's  review  of  the  draft 
revised  protocols,  are  as  follows  (see  Appendix  D  for  full  comments): 

•  Include  interviews  with  direct  care  staff  as  a  major  component  in  reviewing  the 
extent  to  which  individuals  have  individual  support  plans  that  are,  in  fact, 
person  centered,  and  the  extent  to  which  they  are  reaching  their  personally 
desired  outcomes. 

•  Provide  extensive  guidance  (and  related  training)  on  how  goals,  preferences, 
aspirations,  and  so  forth,  are  communicated  by  those  with  severe 
communication  disabilities. 

States  that  have  moved  to  outcome-focused  quality  assurance  systems  report  that  these 
are  areas  that  warrant  particular  attention. 

Reliability  and  Validity 

As  draft  standards  and  related  survey  protocols  are  developed,  we  recommend  that 
pilot  tests  be  conducted  to  assess  reliability  and  validity.  Like  the  analysis  recently 
conducted  by  the  HSRI  project  team  for  HCFA  on  State  outcome-based  quality 
assurance  systems,  tests  should  include  applicability  across  settings  (e.g.,  size  of 
ICF/MR),  concurrent  validity,  and  discriminative  validity,  as  well  as  reliability. 

The  1992  AAMR  Definition  of  Mental  Retardation 

HCFA  may  want  to  consider  referencing  the  new  (1992)  definition  of  mental 
retardation  adopted  by  AAMR,  which  incorporates  the  concept  of  supports  as  an 
integral  component  of  the  assessment  process,  along  with  the  concept  of  five  valued 
accomplishments:  1)  community  presence;  2)  choice,  autonomy  and  control;  3) 
competence,  including  the  opportunity  to  express  one's  gifts  and  capacities;  4)  respect 
and  having  a  valued  place  in  one's  community;  and  5)  community 
participation/network  of  personal  relationships  (Luckasson  and  Spitalnick,  1994).  As 
developed  by  AAMR,  supports  relate  to  these  desired  accomplishments  as  follows: 

Supports  refer  to  an  array,  not  a  continuum,  of  services,  individuals,  and 
settings  that  match  the  person's  needs...[and]  should  be  matched  in  the 
context  of  the  person's  desires.  Supports  are  resources  and  strategies  that 
promote  the  interests  and  causes  of  individuals. ..that  enable  them  to 
secure  access  to  resources,  information  and  relationships  as  part  of 
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inclusive  work  and  living  environments  and  that  result  in  enhanced 
interdependence,  productivity,  community  inclusion,  and  satisfaction. 
Support  resources  are  individual  resources,  skills,  and  competencies  and 
the  ability  and  opportunity  to  make  choices,  manage  money,  manage 
information,  and  the  life.  These  resources  are  also  other  people,  whether 
family,  friends,  coworkers,  people  one  lives  with,  mentors,  or  neighbors. 
Technology  might  also  be  a  form  of  support  resource  and  includes 
assistive  devices,  job  or  living  accommodations,  or  even  behavioral 
technology.  Another  support  resource  encompasses  habitation  services 
that  may  be  needed  if  the  other  naturally  occurring  resources  are  either 
not  available  or  not  sufficient  to  assist  the  person  in  a  desired  living, 
working  or  school  environment.  Clearly,  these  services,  whether 
provided  by  a  specialized  disability  agency  or  a  generic  service  agency, 
are  a  subset  of  supports  (Luckasson  and  Spitalnick,  1994). 

As  indicated  above,  this  is  not  recommended  as  an  alternative  to  the  current  eligibility 
definition  for  ICF/MR  services.  HCFA  may  want  to  consider,  however,  the 
implications  of  revising  the  eligibility  definition  to  include  a  subset  of  those  with 
support  needs.  (See  the  forthcoming  article  in  the  August  1996  issue  of  Mental 
Retardation  for  a  typology  based  on  level  of  intensity  of  supports  needed.) 

Refinements  in  HCFA's  Role  in  ICF/MR  Quality  Assurance 

HSRI  has  suggested  that  HCFA  consider  performance  contracts  with  the  States  in 
ICF/MR  quality  assurance  (see  Task  A  Final  Report).  In  relation  to  active  treatment,  for 
example,  HCFA  standards  could  define  active  treatment,  set  minimum  requirements 
for  its  review,  describe  an  array  of  quality  assurance  components  for  States  to  select  in 
making  ICF/MR  quality  assurance  appropriate  to  their  use  of  the  ICF/MR  program, 
approve  State  ICF/MR  quality  assurance  plans,  and  oversee  compliance  with  these 
plans  through  existing  or  modified  "look  behind"  authority.  The  performance  contract 
approach  also  recognizes  that  States'  ICF/MR  quality  assurance  needs  vary,  for 
example,  States  with  only  a  few  small  ICFs/MR  vs.  States  with  a  few  very  large  State- 
operated  ICFs/MR  or  States  with  hundreds  of  small  ICFs/MR.  This  could  be  coupled 
with  HCFA's  expansion  of  its  role  (recommended  above)  as  a  Center  for  Excellence  in 
quality  assurance,  and  in  sponsoring  research  and  demonstrations  on  effective  quality 
assurance. 

We  suggest  that  HCFA  continue  to  refine  the  definition  of  active  treatment  and  related 
guidelines  based  on  the  recommendations  included  in  this  report  and  that  States  be 
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allowed  to  meet  certification  requirements  either  through  conventional  surveys  or 
through  national  accreditation  that  is  consistent  with  HCFA's  refinements. 

The  overall  recommendations  presented  in  the  separate  Task  A  and  Task  B  reports 
encompass  the  future  regulation  of  active  treatment.  These  include  the 
recommendation  that  HCFA  develop  a  set  of  from  15  to  20  quality  of  life  indicators 
based  on  the  valued  outcomes  and  that  HCFA  take  the  lead  in  developing  and 
publishing  national  norms  for  these  indicators,  through  periodic  national  surveys. 

In  addition,  HSRI  strongly  suggests  that  HCFA  explore  the  application  of  ICF/MR 
quality  assurance  recommendations  on  quality  of  life  indicators,  continuous  quality 
improvement,  external  monitoring  and  publishing  of  quality  assurance  findings  to  the 
Home  and  Community  Based  Services  Waiver  program  (1950(b))  as  well  as  inclusion  in 
the  criteria  for  approval  of  1115  Managed  Care  Waivers. 

Because  of  the  growing  interest  in  Medicaid  managed  care  for  long  term  services  to 
people  with  mental  retardation/developmental  disabilities,  it  is  important  that  HCFA 
explore  ICF/MR  quality  assurance  issues  under  such  a  managed  care  benefit. 
Similarly,  should  States  receive  Medicaid  long  term  care  funds  (or  combined  long  term 
and  acute  care  Medicaid  funds)  in  the  form  of  block  grants,  either  on  a  demonstration 
basis  or  nationwide,  HCFA  will  need  to  provide  guidance  on  the  maintenance  of 
quality  assurance  for  those  currently  in  need  of  the  ICF/MR  level  of  care.  This  will  be 
an  essential  role  for  HCFA  to  play,  regardless  of  any  future  changes  in  their  regulatory 
authority. 
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Appendix  A 

Knowledgeable  Individuals  on  ICF/MR  Quality  Assurance 
Who  Provided  Information  in  Development  of  Task  A  Interim  Report 

ALABAMA 

Kathy  Maddox 

Director  of  Quality  Assurance 

Department  of  Mental  Health  and  Mental  Retardation 

ARIZONA 

Jon  Hash  and  Robert  Harmon 
Program  Operations  Administrators 
Division  of  Developmental  Disabilities 
Department  of  Economic  Security 

ARKANSAS 

Cindy  Horeis 

Quality  Assurance  Program  Evaluator 
Department  of  Human  Services 

CALIFORNIA 

Bonnie  Banks 

Chief,  Quality  Assurance 

Department  of  Developmental  Services 

COLORADO 

Heidi  Cunningham 
Director 

Program  Quality 

Division  for  Developmental  Disabilities 

Kerry  Stern 
Asst.  Director 
Program  Quality 

Division  for  Developmental  Disabilities 
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CONNECTICUT 


Dan  Micari 

Director,  Quality  Assurance 
Department  of  Mental  Retardation 

Fred  Belicki  and  Barbara  Gore 

Quality  Assurance 

Department  of  Mental  Retardation 

DELAWARE 

Eileen  Wray 

Director  of  Quality  Assurance 
Division  of  Mental  Retardation 

FLORIDA 

Marc  Berg  and  Dave  Robinson 

Region  1 ,  Developmental  Disabilities  Services 

Department  of  Health  and  Rehabilitative  Services 

GEORGIA 

Darlene  Meador 
Senior  Psychologist 

Division  of  MH,  MR  &  Substance  Abuse 
HAWAII 

Estella  Bugtong 

Developmental  Disabilities  Division 
Department  of  Health 

IDAHO 

Paul  Swatsenbarg,  Director  and  Frances  Roberts,  Program  Manager 
Developmental  Disabilities  Program 
Department  of  Health  and  Welfare 

Corinne  Horton 

DD  Supervisor  of  Children's  Services 
Department  of  Health  and  Welfare 
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ILLINOIS 


Eranell  Mcintosh-Wilson 
Associate  Director 

Illinois  Department  of  Mental  Health  &  Developmental  Disabilities 
INDIANA 

Judy  Hall,  Survey  Coordinator  and  Mike  Morton,  District  Manager 
Bureau  of  Developmental  Disabilities  Services 
Division  of  Disability,  Aging  &  Rehabilitative  Services 

IOWA 

Ruth  Schanke 

Chief,  Bureau  of  Quality  Assurance  &  Support 
Division  of  MH,  MR  &  DD 

KANSAS 

Susan  Cockerell 

Administrator,  Quality  Enhancement 

Social  and  Rehabilitative  Services/Mental  Health/Rehabilitation  Services 

Gary  Inganthrone 

Quality  Assurance  for  ICFs/MR 

Social  and  Rehabilitative  Services/Mental  Health/Rehabilitation  Services 

KENTUCKY 

Bill  Willson 

Quality  Assurance 

Department  of  Mental  Retardation 

Betty  Weaver 
Branch  Manager 

Department  for  Medicaid  Services 

LOUISIANA 

Bonnie  Butler 

Director,  Division  of  Program  Support 
Department  of  Health  and  Hospitals 
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MAINE 


Linda  Jariz 

Acting  Director 

Division  of  Mental  Retardation 

MASSACHUSETTS 

Mary  Cerreto 

Director,  Office  of  Quality  Enhancement 
Department  of  Mental  Retardation 

MICHIGAN 

Bob  Glees 

Director,  Office  of  Monitoring 
Department  of  Mental  Health 

Judy  Webb 

Federal  Programs  Coordinator 
Department  of  Mental  Health 

MINNESOTA 

Jan  Menke 
Consultant 

Department  of  Human  Services 

MISSISSIPPI 

Ed  LeGrand 
Director 

Bureau  of  Mental  Retardation 
Department  of  Mental  Health 

MISSOURI 

Melinda  Elmore 
Deputy  Director  for  Policy 
Division  of  MR/DD 
Department  of  Mental  Health 
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MONTANA 


Sue  Jackson 

Field  Services  and  Program  Bureau  Chief 
Social  and  Rehabilitative  Services 
Developmental  Disabilities  Division 

NEVADA 

Richard  Tanner 

Director  of  Quality  Assurance 

Sierra  Regional  Center 

NEW  MEXICO 

Arturo  Rangel 

Director,  Office  of  Quality  Assurance 
Developmental  Disabilities  Division 
Department  of  Health 

NEW  YORK 

Tom  Cuite 

Deputy  Director  for  Quality  Assurance 

Office  of  Mental  Retardation  &  Developmental  Disabilities 

Bill  Scanlon 
Quality  Assurance 

Office  of  Mental  Retardation  &  Developmental  Disabilities 
NORTH  CAROLINA 
Fred  Waddle 

Branch  Head,  Residential  and  Vocational  Services 

Developmental  Disabilities  Section 

Division  of  MH,  DD  &  Substance  Abuse  Services 

NORTH  DAKOTA 

Vicci  Pederson 
Quality  Assurance 
Developmental  Disabilities  Division 
Department  of  Human  Services 
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OHIO 


Charles  Grunkemeyer 
Chief,  Office  of  Licensure 
Department  of  MR  &  DD 

Bethanne  Cliffe 

Chief,  Office  of  Supported  Living 
Department  of  MR  &  DD 

Amy  Gerowitz 
CEO 

Williamsburg  Residential  Alternatives 

OKLAHOMA 

Dennis  Bean 

Program  Administrator,  Quality  Assurance  Unit 
Developmental  Disabilities  Services  Division 
Department  of  Human  Services 

OREGON 

Mary  Lee  Fay 

Chief  of  Training/Technical  Assistance 

Office  of  Developmental  Disabilities  Services 

Mental  Health  and  Developmental  Disabilities  Services  Division 

Jimmy  Clark 

Director  of  Quality  Assurance 
Fairview  Training  Center 

RHODE  ISLAND 

Sue  Babin 
Administrator 

Division  of  Developmental  Disabilities 
Department  of  MH,  MR  &  Hospitals 

Tom  Greene 

Habilitation  Services  Manager  -  State  Operated  Facilities 
Division  of  Developmental  Disabilities 
Department  of  MH,  MR  &  Hospitals 
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SOUTH  CAROLINA 


Susan  Thompson 

Director,  Quality  Assurance 

Department  of  Disabilities  &  Special  Needs 

SOUTH  DAKOTA 

Debbie  Bowman 
Director 

Division  of  Developmental  Disabilities 

TEXAS 

Sue  Dillard 

Director  of  Quality  Assurance 

Department  of  Mental  Health/Mental  Retardation 

Catherine  Hayes 
QMRP/Administrator 
Austin  State  School 

UTAH 

Georgia  Baddley 

Director,  Quality  Assurance  Program 

Division  of  Services  for  People  with  Disabilities 

Department  of  Human  Services 

VERMONT 

Carol  Leach 

Community  Alternatives  Specialist 
Division  of  Mental  Retardation 

WASHINGTON 

Maureen  Weeks 

Quality  Management  Program  Manager 
Division  of  Developmental  Disabilities 
Department  of  Social  &  Health  Services 
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WEST  VIRGINIA 


Marc  Hannah 

Director  of  Long  Term  Care 

WYOMING 

Jon  Fortune 

Adult  Services  Manager 

Division  of  Developmental  Disabiliti 

Department  of  Health 

Dave  Bruner 

Director  of  Quality  Assurance 
Wyoming  State  Training  School 
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EVALUATION  OF  ICFs/MR 


BEHAVIOR  DEVELOPMENT  SURVEY 

Developed  by: 

Temple  University 
Institute  on  Disabilities/UAP 
November  13, 1995 


Please  Print 

Sequence  #   

Person's  Social  Security  #   


Person ' s  Name 

Last  First  M.I. 


Residential  Provider 
Complete  Site  Mailing  Address 
City,   Town  State  Zip  Code 

Primary  Respondent  Name 

J  )  

Telephone 


Name  of  Data  Collector 
Page  1 


DEMOGRAPHICS 

1.       TODAY'S  DATE 

/ 


Month  Year 

2.       PERSON'S  DATE  OF  BIRTH 

/  / 


Month  Day  Year 

RESIDENTIAL  PLACEMENT  TYPE 


1  Public  Institution  (Not  ICF/MR) 

2  ICF/MR,  1 6  beds  or  more 

3  ICF/MR,  4  to  15  beds 

4  Community  Living  Arrangement,  group 
home  or  apartment  (30  hours  or  more  of 
staff  time  per  week) 

5  Community  Living  Arrangement,  group 
home  or  apartment  (Less  than  30  hours  of 
staff  time  per  week) 

6  Supported  Living  (30  hours  or  more  of  staff 
time  per  week) 

7  Supported  Living  (Less  than  30  hours  of 
staff  time  per  week) 

8  Home  and  Community  Based  Waiver 
Services  (in-home) 

9  Foster  Family,  Family  Living  Program 

10  With  Family  or  in  own  Home 

1 1  Nursing  Home 

12  Other  (Describe)   


4.       LEVEL  OF  RETARDATION 
(Please  seek  documentation) 


5  Profoundly  retarded 

4  Severely  retarded 

3  Moderately  retarded 

2  Mildly  retarded 

1  Not  mentally  retarded 


2 


OTHER  DISABILITIES 

Seek  documentation.  Enter  code  in  box.  Leave 
none  blank. 

1  Yes 
0  No 

9    Don't  know 


Autism 


Blind 


Mental  Disorder 


Other  Health  Impairment  (specify): 


Cerebral  Palsy 


Epilepsy,  Controlled 


Epilepsy,  Uncontrolled 


Non-Ambulatory,  Mobile 


Non-Ambulatory,  Non-Mobile 


6.  SEX 


1  Male 

2  Female 


Speech/Language  Impairment 


Hard  of  Hearing 


RACE 


1  White 

2  African-American 

3  Latino 

4  Asian 

5  Other 


Deaf 


Visual  Impairment 


HEALTH  AND  SAFETY 

MEDICATION: 

1 .       How  many  medications  does  this 

individual  take  on  a  daily  basis?  Count 
each  medication  that  is  given  to  this 
person  daily  and  that  is  prescribed.  Do 


3 


Has  the  person  been  screened  for  side  effects 
of  psychotropic  medications?  (Answer  yes  only 
if  AIMS  or  DISCUS  screenings  are 
documented). 


not  count  vitamins  or  topical  ointments.  4- 
Enter  the  exact  number. 

  □ 

Is  this  individual  receiving  any 
medication  for  the  purpose  of  behavior 
change? 


1  Yes 

2  No 


If  this  individual  is  receiving  medication  for  the 
purpose  of  behavior,  has  the  prescribing 
physician  certified  the  continuing  medication 
within  the  past  90  days? 

1  Yes 

2  No 
9  NA 


1  Yes 

2  No 

9  NA  -  person  receives  no 
psychotropic  medications. 

5.  Results  of  medication  screening. 

Does  this  person  have  any  side  effects 
resulting  from  medication  as 
documented  by  the  above  listed 
screening  tool?  (AIMS  or  DISCUS) 


2  No 

8  Unknown-AIMS  or  DISCUS 
not  administered 

9  NA  -  person  receives  no 
psychotropic  medications. 


□ 


□ 


If  this  person  receives  psychotropic 
medication,  does  he/she  have  a 
psychiatric  diagnosis? 

1  Yes 

2  No 

9  NA  -  person  receives  no 
psychotropic  medications. 


If  this  person  receives  psychotropic 
medication,  does  he/she  have  a  formal 
behavior  plan  which  seeks  to  reduce  the 
use  of  this  medication  overtime? 


1  Yes 
2  No 

9     NA-person  receives  no 

psychotropic  medication 


4 


MEDICAL  NEEDS: 


□ 


In  general,  how  urgent  is  this  person's  need  for 
medical  care? 


3  Generally  has  no  serious  medical  needs 
2    Needs  visiting  nurse  and/or  regular  visits 

to  the  doctor 
1    Has  life-threatening  condition  that  requires 

very  rapid  access  to  medical  care 
0    Would  not  survive  without  24-hour 

medical  care 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  probtem(s) 

Describe  the  problem: 


2.  How  many  times  has  this  person  seen  each  of 
the  following  doctors  in  the  past  year? 

  a.  General  Practitioner/Internist 

  b.  Obstetrics/Gynecologist 

  c.  Psychiatrist 

  d.  Dentist 

  e.  Podiatrist 

  f.  All  others 


3.       Has  there  been  a  problem  getting  medical  care 
during  the  past  year? 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  problem(s) 

Describe  the  problem: 


4.       Has  there  been  a  problem  getting  dental  care 
during  the  past  year? 


5.       Has  there  been  a  problem  getting  mental  health 
care  during  the  past  year? 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  problem(s) 

Describe  the  problem: 


.  f.  Fire  requiring  fire  department 

.  g.  Emergency  program  relocation 

.  h.  Alleged  misuse  of  client  funds  or  property 

.  i.  Outbreak  of  communicable  disease 

.  j.  Alleged  violation  of  client  rights 


k.    Other  (please  specify) 


6.       Does  this  person  have  an  Individual  Health  Care 
Plan? 


1  Yes 

2  No 

UNUSUAL  INCIDENT  REPORTS 

1 .         Have  there  been  any  unusual  incidents 

involving  this  individual  in  the  past  6  months 
that  you  have  reviewed? 


1  Yes 


2  No 

2.         If  yes,  how  many?  (Enter  exact  number) 


4.  For  the  unusual  incidents  you  identified,  please 

indicate  how  many  of  each  type  were  reported  . 

a.  Rape  (alleged)   Victim 

Rape  (alleged)   Perpetrator 

b.  Assault  (alleged)   Victim 

Assault  (alleged)   Perpetrator 

c.  Abuse  (alleged)  Victim 

Abuse  (alleged)  Perpetrator 

d.  Serious  behavior  incident:  police  involvement 
with  a  client  specific  incident 

 Victim 

 Perpetrator 

e.  Serious  behavior  incident:  emergency 

restraint 

 Victim 

 Perpetrator 


3.   For  the  unusual  incidents  you  identified,  please 
indicate  how  many  of  each  type  were  reported. 


a. 

Injury/illness  requiring  hospital  admission 

b. 

Serious  illness  requiring  doctor 

c. 

Medication  error 

d. 

Suicide  attempt 

e. 

Unauthorized  absence  -  immediate  jeopardy  or 

more  than  24  hours 

5.       Are  there  any  instances  of  abuse,  neglect,  or 

mistreatment  that  have  gone  untreated?  (Please 
look  at  1  month's  daily  logs). 

I  1  1  Yes 

2  No 

If  yes,  please  describe: 
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BEHAVIOR  PLAN 


1 .      Does  this  person  have  a  formal  behavior  plan? 


Yes 
2  No 


□ 


□ 


If  the  answer  to  question  4  is  yes,  is  the  use  of 
the  Time-Out  Room  part  of  a  program  plan? 

1  Yes 

2  No 
9  NA 


Is  there  any  evidence  of  the  use  of  painful 
and/or  noxious  stimuli? 


1  Yes 

2  No 


If  yes,  when  did  the  use  occur? 


2. 


Is  there  any  evidence  of  use  of  physical 
restraints? 


1  Yes 

2  No 

If  yes,  when  did  they  occur? 


7. 


□ 


If  the  answer  to  question  6  was  yes,  is  the  use  of 
painful  and/or  noxious  stimuli  part  of  a  program 
plan? 

1  Yes 

2  No 
9  NA 


8.       Are  there  any  unusual  incidents  that  have  not 
If  the  answer  to  question  2  is  yes,  are  the  use  of  been  reported?  (Look  for  documentation) 

restraints  part  of  a  program  plan?  ^  j 

Yes 
2  No 

9  Describe  the  incident: 


1  Yes 

2  No 


Is  there  any  evidence  of  use  of  Time-Out 
Rooms? 


Yes 
2  No 


If  yes,  when  did  the  use  occur? 


SAFETY 


□ 


Are  there  adequate  number  of  smoke  detectors 
in  this  house/apartment  /living  area  (one  per 
floor  including  attic  and  basement)? 

1  Yes 

2  No 
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8.Are  there  any  physical  hazards  within  this  living  area? 


2.      Do  all  of  the  smoke  detectors  work,  or  does  the 


3.      Are  all  of  the  fire  extinguishers  properly 
charged? 

1  Yes 

2  No 


4.      Was  there  a  fire  drill  in  the  past  60  days? 


1  Yes 

2  No 

9  NA  (minimal  supervision) 


5.      Was  there  a  fire  drill  in  the  60  days  before  that? 


1  Yes 

2  No 

9  NA  (minimal  supervision) 


6.       Is  this  person's  living  area  reasonably  clean? 


1  Yes 

2  No 

If  no,  please  explain: 


7.  Are  there  any  excessively  unpleasant  odors  in  this 
living  area? 

1  Yes 

2  No 

If  yes,  please  explain: 


system  work? 


1  Yes 

2  No 


1  Yes 

2  No 


If  yes,  please  explain: 
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9.         Does  this  individual  have  ail  prescribed  adaptive 
equipment?  (Base  your  judgment  on  the 
individual  plan  as  well  as  any  other 
documentation  such  as  prescriptions,  etc.) 

1  Yes 

2  No 

If  No,  please  describe: 


1 0.      Are  all  living  and  program  areas  accessible  to 
this  individual? 


1  Yes 

2  No 


1 1 .       Does  this  individual  have  an  Individualized 
Safety  Plan? 

1  Yes 

2  No 


1 2.       How  safe  is  this  neighborhood? 

5  Extremely  safe 

4  Safe 

3  Neither  safe  nor  unsafe 

2  Unsafe 

1  Extremely  unsafe 


13.1s  your  neighborhood  accessible  in  a  way  that  meets 

your  needs  (e.g.  ramps,  curb  cuts,  audible  traffic 
signals,  etc.)? 

3    Completely  accessible 
2    Somewhat  accessible 
1    Not  accessible  at  all 


PLANNING 


1.        PLANNING  DOCUMENT 

Is  there  a  planning  document  (IHP/IPP)  that  is 
current  (within  one  year)  for  this  person  on  site? 

1  Yes 

2  No 


CASE  MANAGEMENT/ 
SERVICE  COORDINATION 


1 .        Is  there  someone  who  acts  on  the 
person's  behalf  as  the  Individual 
Service  Coordinator/Case  Manager  who 
provides  service  coordination  for  the 
individual;  advocates  on  his/her  behalf; 
provides  networking  and  monitors  the 
provision  of  support  for  this  individual? 


1  Yes 

2  No 


DIGNITY 

1 .       Does  the  individual  have  personal 

possessions  (e.g.  tv,  stereo,  pictures, 
furniture,  bicycle,  knick  knacks)?  - 


1  Yes 

2  No 

If  No,  please  explain: 
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2.      Does  this  individual  have  sufficient 
personal  storage  space?  (e.g.  closet 
space  and  a  chest  of  drawers) 


1  Yes 

2  No 

If  No,  please  explain: 


3.        Does  this  individual  have  an  adequate 
wardrobe,  in  both  repair  and  fit? 


1  Yes 

2  No 


If  No,  please  explain: 


4.  Is  the  person  dressed  appropriately? 


1  Yes 

2  No 


If  No,  please  explain: 


5. 


Is  the  person  clean  and  groomed 
appropriately? 


1  Yes 

2  No 

If  No,  please  explain: 


6.        Is  this  person  treated  with  respect? 


1  Yes 

2  No 

If  No,  please  explain: 


SELF  DETERMINATION 

1 .        What  degree  of  freedom  does  this 
person  have  to  make  important 
decisions  by  him/herself  which 
influence  him/her  everyday  life? 
(Please  circle  your  response  below) 


1  2  3 


About  the  same 
degree  of  freedom 
as  you  and  I 


Some  degree  of 
freedom  over 
everyday  life 


4  5 

No  degree 

of  freedom 


Tossebro,  Jan  (1995).  Impact  of  Size  Revisited:  Relation 
of  Number  of  Residents  to  Self- Determination  and 
Deprivatization.  American  Journal  on  Mental 
Retardation,  1995, 100,59-67. 


ADAPTIVE  BEHAVIOR 

•  GIVE  CREDIT  ONLY  FOR  WHAT  THE  PERSON 
DOES  DO.  NOT  FOR  WHAT  THE  PERSON 
"CAN"  DO  OR  "COULD"  DO  OR  "MIGHT  BE 
ABLE  TO"  DO.  WE  WANT  NO  SPECULATION  - 
ONLY  OBSERVED,  ACTUAL  BEHAVIORS. 

.     IF  THE  BEHAVIOR  IS  PERFORMED  WITH 
VERBAL  PROMPTS,  GIVE  CREDIT  (UNLESS 
OTHERWISE  NOTED  IN  THE  ITEM).  DO  NOT 
GIVE  CREDIT  FOR  BEHAVIORS  PERFORMED 
WITH  PHYSICAL  GUIDANCE  (UNLESS 
OTHERWISE  NOTED  IN  THE  ITEM). 

•  GIVE  CREDIT  FOR  A  BEHAVIOR  IF  IT  IS 
PERFORMED  AT  LEAST  75%  (THREE 
FOURTHS)  OF  THE  TIME. 


•  ON  ANY  ITEM,  A  ZERO  CAN  BE  ENTERED  IF 
THE  ITEM  IS  NOT  APPLICABLE,  OR  IF  THE 
PERSON  IS  TOO  YOUNG,  UNABLE,  OR  HAS 
NO  OPPORTUNITY  TO  DISPLAY  THE 
BEHAVIOR. 

•  ON  ITEMS  LIKE  #1 ,  CHECK  ALL  THAT  APPLY, 
ADD  UP  THE  NUMBER  OF  CHECKS,  AND 
ENTER  THAT  NUMBER  IN  THE  BOX. 

1 .       WALKING  AND  RUNNING  (Check  ALL  that 
apply.  With  cane,  crutches,  brace,  or  walker,  if 
used.) 


 a  Walks  alone 

 b   Walks  up  and  down  stairs  alone 

 c   Walks  down  stairs  by  alternating  feet 

 d    Runs  without  falling  often 

 e   Hops,  skips  or  jumps 

(NONE  OF  THE  ABOVE:  ENTER  0) 

2.       BODY  BALANCE 


5  Stands  on  "tiptoe"  for  ten  seconds  if  asked 

4  Stands  on  one  foot  for  two  seconds  if  asked 

3  Stands  without  support 

2  Stands  with  support 

1  Sits  without  support 

0  Can  do  none  of  the  above 

3.  USE  OF  TABLE  UTENSILS 


6    Uses  knife  and  fork  correctly  and  neatly 
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5  Uses  table  knife  for  cutting  or  spreading 

4  Feeds  self  with  spoon  and  fork  -  neatly 

3  Feeds  self  with  spoon  and  fork  - 

considerable  spilling 

2  Feeds  serf  with  spoon  -  neatly 

1  Feeds  self  with  spoon  -  considerable  spilling 

0  Feeds  self  with  fingers  or  must  be  fed 


EATING  IN  PUBLIC  (Visual  aids  are 
acceptable) 


3  Orders  complete  meals  in  restaurants 
2    Orders  simple  meals  Ike  hamburgers 

or  hot  dogs 
1    Orders  soft  drinks  at  soda  fountain  or 

canteen 

0  Does  not  order  food  at  public  eating  places 
DRINKING 


3    Drinks  without  spilling,  holds  glass  in  one 
hand 

2    Drinks  from  cup  or  glass  unassisted  -  neatly 
1    Drinks  from  cup  or  glass  -  considerable 
spilling 

0    Does  not  drink  from  cup  or  glass 
TOILETING 


4    Never  has  toilet  accidents 

3    Never  has  toilet  accidents  during  the  day 

2    Occasionally  has  toilet  accidents  during  the 

day 

1  Frequently  has  toilet  accidents  during  the  day 
0    Is  not  toilet  trained  at  all 

SELF-CARE  AT  TOILET  (Check  ALL  that  apply) 


Lowers  pants  at  the  toilet  without  help 
Sits  on  toilet  seat  without  help 
Uses  toilet  tissue  appropriately 
Rushes  toilet  after  use 
Puts  on  clothes  without  help 
Washes  hands  without  help 
OF  THE  ABOVE:  ENTER  0) 

WASHING  HANDS  AND  FACE  (Check  ALL  that 
apply) 


Washes  hands  with  soap 

Washes  face  with  soap 

Washes  hands  and  face  with  water 

Dries  hands  and  face 

OF  THE  ABOVE:  ENTER  0) 


 b 

 c 

 d 

 e 

 f 

(NONE 


 b 

 c 

 d 

(NONE 


BATHING 


13.       SENSE  OF  DIRECTION 


6    Prepares  and  completes  bathing  unaided 
5   Washes  and  dries  self  completely 
4   Washes  and  dries  reasonably  well  with 

prompting 
3    Washes  and  dries  self  with  help 
2    Attempts  to  soap  and  wash  self 
1    Cooperates  when  being  washed  and  dried 

by  others 

0    Makes  no  attempt  to  wash  or  dry  self 
CARE  OF  CLOTHING  (Check  ALL  that  apply) 


 a  Cleans  shoes  when  needed 

 b   Puts  clothes  in  drawer  or  chest  neatly 

 c   Puts  soiled  clothes  in  proper  place  for 

laundering/washing,  without  being 

reminded 

 d   Hangs  up  clothes  without  being 

reminded 
(NONE  OF  THE  ABOVE:  ENTER  0) 

DRESSING 


Goes  several  blocks  from  grounds,  or  from 

home,  without  getting  lost 

Goes  around  grounds  or  a  couple  of  blocks 

from  home  without  getting  lost 

Goes  around  cottage,  ward,  yard,  or  home 

without  getting  lost 

Demonstrates  no  sense  of  direction 


14.MONEY  HANDLING 


15. 


Uses  money  with  little  or  no  assistance 
(e.g.,  assistance  with  budgeting  is  OK) 
Uses  money  with  minor  assistance 
(e.g.,  checking  for  correct  change,  etc.) 
Uses  money  with  some  assistance 
(e.g.,  being  told  the  correct  bills  or  coins) 
Uses  money  with  complete  assistance  of 
staff 

Does  not  use  money 


PURCHASING 


5   Completely  dresses  self 

4    Completely  dresses  self  with  verbal 

prompting  only 
3    Dresses  self  by  pulling  or  putting  on  all 

clothes  with  verbal  prompting  and  by 

fastening  (zipping,  buttoning,  snapping) 

them  with  help 
2    Dresses  self  with  help  in  pulling  or  putting 

on  most  clothes  and  fastening  them 
1    Cooperates  when  dressed,  e.g.,  by 

extending  arms  or  legs 
0    Must  be  dressed  completely 


5  Chooses  and  buys  all  own  clothing  without  help 

4  Chooses  and  buys  some  clothing  without  help 

3  Makes  minor  purchases  without  help 

(e.g.,  snacks,  drinks) 

2  Does  some  shopping  with  slight  supervision 

1  Does  some  shopping  with  close  supervision 

0  Does  no  shopping 


16.  WRITING 


12.   SHOES  (Check  ALL  that  apply) 

 a   Puts  on  shoes  correctly  without 

assistance 

 b  Ties  shoe  laces  without  assistance 

(Velcro  does  not  count) 

 c   Unties  shoe  laces  without  assistance 

 d   Removes  shoes  without  assistance 

(NONE  OF  THE  ABOVE:  ENTER  0) 


Writes  complete  lists,  memos,  or  letters 
Writes  short  sentences 
Writes  or  prints  more  than  ten  words  without 
copying  or  tracing 

Writes  or  prints  own  name  or  other  words 

without  copying  or  tracing 

Traces  or  copies  own  name  or  other  words 

Does  not  write,  print,  copy,  or  trace  any 

words 
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17.  PREVERBAL  EXPRESSION  (Check  ALL  that  apply) 


 a   Is  able  to  say  (sign)  at  least  a  few  words 

(If  so,  enter  a  "6"  in  the  circle) 

 b  Nods  head  or  smiles  to  express 

happiness 

 c   Indicates  hunger 

 d   Indicates  wants  by  pointing  or  vocal 

noises 

 e   Expresses  pleasure  or  anger  by  vocal 

noises 

 f    Chuckles  or  laughs  when  happy 

(NONE  OF  THE  ABOVE:  ENTER  0) 


18.SENTENCES 


3    Sometimes  uses  complex  sentences 
containing  "because,"  "but,"  etc. 

2    Asks  questions  using  words  such  as  "why," 
"how,"  "what,"  etc. 

1    Speaks  in  simple  sentences 

0    Is  non-verbal  or  nearly  non-verbal 


19.  READING 


5    Reads  books  or  other  materials  suitable  for 

children  9  years  old  or  older 
4    Reads  books  or  other  materials  suitable  for 

children  7  years  old 
3    Reads  simple  stories  or  comics  suitable  for 

kindergarten  or  1st  grade  level 
2  Recognizes  1 0  or  more  words 
1    Recognizes  various  signs,  such  as  "EXIT"  or 

"STOP"  or  "WOMEN"  or  "MEN"  or  street 

signs. 

0    Recognizes  no  words  or  signs. 


20.  COMPLEX  INSTRUCTIONS  (Check  ALL  that  apply) 

 a   Understands  instructions  containing 

prepositions,  e.g.,  "on,"  "in,"  "behind" 

 b   Understands  instructions  referring  to  the 

order  in  which  things  must  be  done, 
e.g.,  "first  do  this,  and  afterward, 
do  that." 

 c    Understands  instructions  requiring  a 

decision,  e.g.,  "If  there's  any  ham,  make 
(NONE  OF  THE  ABOVE:  ENTER  0) 


21.  NUMBERS 


8  Does  simple  addition  and/or  subtraction 

4  Counts  1 0  or  more  objects 

3  Mechanically  counts  aloud  from  1  to  10 

2  Counts  2  objects  between  saying  "one,  two" 

1  Discriminates  between  "one"  and  "many" 

0  Has  no  understanding  of  numbers 


22.        TIME  (Check  ALL  that  apply) 


 a   Tells  time  by  clock  or  watch  correctly 

 b   Understands  time  intervals,  e.g.,  there 

is  1  hour  between  3:30  and  4:30 
 c   Understands  timeequivalents,  e.g., " 

9:15"  is  the  same  as  "quarter  past  nine" 
 d   Associates  time  on  clock  with  various 

actions  and  events,  e.g.,  6:00  means 

dinner  time 
(NONE  OF  THE  ABOVE:  ENTER  0) 
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23.         ROOM  CLEANING 

2    Cleans  room  well,  e.g.,  sweeping, 

vacuuming,  tidying 
1    Cleans  room  but  not  thoroughly 
0    Does  not  clean  room  at  all 


24.         FOOD  PREPARATION 

3    Prepares  an  adequate  complete  meal  (may 

use  canned  or  frozen  foods) 
2   Mixes  and  cooks  simple  foods,  e.g.,  fries 

eggs,  makes  pancakes,  cooks  TV  dinners, 

can  of  soup 
1    Prepares  simple  foods  requiring  no  mixing 

or  cooking,  e.g.,  sandwiches,  cold  cereal,  etc. 
0    Does  not  prepare  food  at  all 

25.       TABLE  CLEARING 


2    Clears  table  of  breakable  dishes  and 
glassware 

1    Clears  table  of  unbreakable  dishes  and 

silverware 
0    Does  not  clear  table  at  all 


26.       JOB  COMPLEXITY 


2  Competitive  employment  or  goes  to  workshop 
1  In  pre- vocational  training,  in  school,  or  retired 
0    Performs  no  work 


27.  INITIATIVE 


3    Initiates  most  of  own  activities 
2    Initiates  some  of  own  activities 
1    Will  engage  in  activities  only  if  assigned  or 
directed 

0    Will  not  engage  in  assigned  activities 
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28.  ATTENTION 


4   Will  pay  attention  to  purposeful  activities  for 

more  than  20  minutes 
3    Will  pay  attention  to  purposeful  activities  for 

about  15  minutes 
2    Will  pay  attention  to  purposeful  activities  for 

about  10  minutes 
1    Will  pay  attention  to 

0    Will  not  pay  attention  to  purposeful  activities 
for  as  long  as  5  minutes 

29.       PERSONAL  BELONGINGS 


3    Very  dependable,  always  takes  care  of 
belongings 

2    Usually  dependable,  usually  takes  care  of 
belongings 

1    Unreliable,  seldom  takes  care  of  belongings 
0    Not  responsible  at  all,  does  not  take  care  of 
belongings 


30.       AWARENESS  OF  OTHERS  (Check  ALL  that 
apply) 


 a   Recognizes  own  family 

 b   Recognizes  people  other  than  family 

(If  "b"  is  checked,  also  check  "a") 
 c   Has  information  about  others,  e.g., 

relation  to  self,  job,  address,  name 
 d   Knows  the  names  of  people  close  to 

him/her,  e.g.,  in  neighborhood,  at  home 

or  day  program 
 e   Knows  the  names  of  people  not 

regularly  encountered 
(NONE  OF  THE  ABOVE:  ENTER  0) 


31 .       INTERACTION  WITH  OTHERS 


3    Interacts  with  others  for  more  than  5 
minutes 

2    Interacts  with  others  for  up  to  5  minutes 
1    Interacts  with  others  in  limited  ways,  e.g., 

eye  contact,  handshakes,  responsive  to 

touch 

0    Does  not  interact  with  others  in  a  socially 
acceptable  manner 

32.       PARTICIPATION  IN  GROUP  ACTIVITIES 


3    Initiates  group  activities  at  least  some  of  the 

time  (leader  and/or  organizer) 
2    Participates  in  group  activities 

spontaneously  and  eagerly  (active 

participant) 

1    Participates  in  group  activities  rf  encouraged 

to  do  so  (passive  participant) 
0    Does  not  participate  in  group  activities 

(unless  physically  guided) 


CHALLENGING  BEHAVIOR  IN  THE  PAST  4  WEEKS: 


FREQUENCY 

2       Not  observed  in  the  past  4  weeks 
1       5  times/week  or  less  in  past  4  weeks 
0       More  than  5  times/week  in  past  4  weeks 


1 .     Threatens  or  does  physical  violence  to  others 


2.     Damages  own  or  others'  property 


3.     Disrupts  others'  activities  (intentionally) 


4.     Uses  profane  or  hostile  language 


5.     Is  rebellious,  e.g.,  ignores  regulations,  resists 
following  instructions 


Runs  away  or  attempts  to  run  away 
(intentionally) 


Is  untrustworthy,  e.g.,  takes  others'  property, 
lies,  or  cheats 


1 2.     Socially  unacceptable  sexual  behavior 


13.     Listless,  sluggish,  inactive,  unresponsive  to 
activities 


14.     Screams,  yells,  or  cries  inappropriately 


1 5.     Repeats  a  word  or  phrase  over  and  over 


1 6.     Threatens  violence  to  self 


8.     Displays  stereotyped  behavior,  e.g.,  rocks 
body,  hands  constantly  moving  in  repetitive 
pattern 


9.     Removes  or  tears  off  own  clothing 
inappropriately 


1 0.     Injures  self 


11.     Is  hyperactive,  e.g.,  will  not  sit  still  for  any 
length  of  time 
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SEVERITY 


No  problem 
Minor  problem 
Major  problem 

Extremely  urgent  problem,  completely  or  nearly 
intolerable 


1 7.     Threatens  or  does  physical  violence  to  others 


1 8.     Damages  own  or  others'  property 


19.     Disrupts  others'  activities  (intentionally) 


20.     Uses  profane  or  hostile  language 


21 .     Is  rebellious,  e.g.,  ignores  regulations,  resists 
following  instructions 


22.     Runs  away  or  attempts  to  run  away 
(intentionally) 


23.     Is  untrustworthy,  e.g.,  takes  others'  property, 
lies,  or  cheats 


24.     Displays  stereotyped  behavior,  e.g.,  rocks 
body,  hands  constantly  moving  in  repetitive 
pattern 


25.     Removes  or  tears  off  own  clothing 
inappropriately 


26.     Injures  self 


27.     Is  hyperactive,  e.g.,  will  not  sit  still  for  any 
length  of  time 


28.     Socially  unacceptable  sexual  behavior 


29.     Listless,  sluggish,  inactive,  unresponsive  to 
activities 


30.     Screams,  yells,  or  cries  inappropriately 


31 .     Repeats  a  word  or  phrase  over  and  over 


32.     Threatens  violence  to  self 


33.       Did  this  person  display  any  serious  challenging  behavior  that  did  not  happen  in  the  past  four  weeks  that  is  of  serious 
concern  to  staff?  Describe: 


34.       Does  this  person  have  a  severe  reputation?  If  yes,  what  is  it  and  what  impact  does  it  have  on  this  person's  life? 
Describe: 
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COMMUNITY  INTEGRATION  /  INCLUSION 
SOCIAL  PRESENCE  IN  THE  PAST  WEEK  (Copyright  CFA  1987) 


We  define  SOCIAL  PRESENCE  as  the  frequency  of  "CONTACT"  with  PEOPLE  WITHOUT  DISABILITIES  WHO  DO  NOT  LIVE 
WITH,  AND  WHO  ARE  NOT  PAID  TO  INTERACT  WITH,  this  individual.  People  who  are  paid  to  interact  with  this  individual  include 
residential  and  day  program  staff,  case  managers,  doctors,  physical  and  occupational  therapists,  etc. 

"CONTACT"  is  defined  as: 

An  event  of  more  than  5  minutes  in  which  the  individual: 

Interacts  with,  or 
Is  seen  by,  or 

Is  in  the  presence  of  people  without  disabilities. 


For  each  category  below,  enter  the  number  of  times  "CONTACT"  has  occurred  for  this  person  in  the  PAST  WEEK. 

Limit  the  number  of  contacts  to  5  per  day  or  35  per  weak  In  any  category.  If  the  person  is  in  supported  or  competitive 

employment,  score  a  maximum  of  5  contacts  for  each  day  at  work.  Riding  the  bus  one  way  equals  one  contact. 

GROUP  SIZE  HOW  MANY  OF 

(NUMBER  OF  PEOPLE  WITH  DISABILITIES  THESE  CONTACTS 

WHO  WERE  PRESENT)  INVOLVED 

COMMUNICATION 

1  2  to  5  6  or  MORE 

SETTING  TYPE 

IN  OWN  OR  OTHER  PERSON'S  HOME,  OR 

IN  OWN  OR  OTHER  PERSON'S  NEIGHBORHOOD     

1.  2. 

RECREATIONAL     

5.  6. 

COMMERCIAL     

9.  10 

DAY  PROGRAM/JOBS     

13.  14 

DURING  TRANSPORTATION     

17.  18 

21. 


Check  here  it  there  has  not  been  any  contact  with  people  without  disabilities  in  the  past  week. 


HARRIS  SCALE  OF  INTEGRATED  ACTIVITIES 

22.       About  how  often  did  this  person  do  each  of  the  following  in  the  past  year: 


a.  Visit  with  close  friends,  relatives  or  neighbors 

b.  Visit  a  supermarket  or  food  store 

c.  Go  to  a  restaurant 

d.  Go  to  church  or  synagogue 

e.  Go  to  a  shopping  center,  mall  or  other  retail  store  to  shop 

f .  Go  to  bars,  taverns,  etc. 
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3. 


4. 


7. 


8. 


11. 


12. 


15. 


16. 


19. 


20. 


g.    Go  to  the  bank 


1  =  More  than  twice  a  week 

2  =  Twice  a  week 

3  =  Once  a  week 

4  =  2-3  times  a  month 

5  =  Once  a  month 

6  =  Less  than  once  a  month 

7  =  Never 

9  =  Not  sure  or  refused 


CONSUMER  SATISFACTION 
INTERVIEW  WITH  PERSON  (Copyright  CFA  1986) 


These  questions  should  be  answered  by  the  consumer,  and  if  appropriate  and  feasible,  should  be  answered  in  private. 

Ask  the  person  if  he/she  is  willing  to  talk  to  you.  If  not,  do  not  proceed;  enter  all  O's. 

A  zero  (0)  should  be  used  when  an  item  is  not  applicable,  or  for  which  the  person  is  too  young  or  unable  to 

respond,  or  if  no  answer  is  given. 

Ask  the  person  if  it  is  OK  to  tell  others  what  you  are  telling  me.  If  not,  do  not  record  verbatim  responses. 


□ 


DO  YOU  LIKE  THE  FOOD  HERE? 
3    Yes,  like  food 

2    In  between,  sort  of,  unsure,  etc. 
1    No,  do  not  like  food 
0    Did  not  answer 


2.       DO  YOU  LIKE  LIVING  HERE? 

□ 

3  Yes,  like  living  here 

2  In  between,  sort  of,  unsure,  etc. 

1  No,  do  not  like  living  here 

0  Did  not  answer 


3.       PLEASE  LET  ME  CHECK  -  DID  YOU  SAY  THE  FOOD  HERE  IS  BAD  OR  GOOD? 

□ 

3  Good 

2  In  between,  sort  of,  unsure,  etc. 

1  Bad 

0  Did  not  answer 


4.       DO  YOU  LIKE  THE  PEOPLE  WHO  WORK  HERE? 

□ 
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3  Yes 

2  Somewhat,  most,  not  all,  etc. 

1  No 

0  Did  not  answer 

5.  DO  YOU  HAVE  ENOUGH  CLOTHES  TO  WEAR? 

□ 

3  Yes,  enough  clothes 

2  In  between,  sort  of,  unsure,  etc. 

1  No,  not  enough  clothes 

0  Did  not  answer 

6.  DO  YOU  HAVE  ANY  REAL  GOOD  FRIENDS? 

□ 

4  Yes,  more  than  one 

3  Yes,  one 

2  In  between,  sort  of,  unsure,  etc. 

1  No,  no  real  good  friends 
0  Did  not  answer 


7.  ARE  PEOPLE  WHO  WORK  HERE  MEAN  OR  NICE? 

3  Nice 

2  In  between,  unsure,  etc. 

1  Mean 

0  Did  not  answer 

8.  DO  YOU  LIKE  WHAT  YOU  DO  DURING  THE  DAY?  (JOB,  DAY  PROGRAM,  ETC.) 

□ 

3  Yes 

2  In  between,  sort  of,  unsure,  etc. 

1  No 

0  Did  not  answer 

9.  DO  YOU  MAKE  ANY  MONEY? 

□ 

3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


1 0.  IS  IT  ENOUGH  TO  MEET  YOUR  NEEDS? 

□ 

3  Yes 

2    Unsure,  etc. 
1  No 

0    Did  not  answer 

11.  DO  YOU  KNOW  WHAT  MEDICATION  YOU  TAKE? 
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□ 


3  Yes 

2   Unsure,  etc. 
1  No 

0    Did  not  answer 
12.D0ES  THE  MEDICATION  YOU  TAKE  HELP  YOU? 


3  Yes 

2    Unsure,  etc. 
1  No 

0    Did  not  answer 

13.       DO  YOU  HAVE  A  CHOICE  ABOUT  WHAT  MEDICATION  YOU  TAKE? 


3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


14. DO  YOU  HAVE  ENOUGH  PRIVACY  HERE? 


3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


1 5.       DO  YOU  KNOW  WHO  YOUR  CASE  MANAGER  IS? 


3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


1 6.       DO  YOU  SEE  YOUR  CASE  MANAGER  AT  LEAST  ONCE  EVERY  MONTH? 


3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


17.       DID  YOU  HELP  DEVELOP  YOUR  PLAN? 
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3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


1 8.       DO  YOU  AGREE  WITH  EVERYTHING  IN  YOUR  PLAN? 

□ 

3  Yes 

2  Unsure,  etc. 

1  No 

0  Did  not  answer 


19.       IS  THERE  ANYTHING  ELSE  THAT  YOU  WOULD  LIKE  TO  TELL  ME? 
(Record  response  word  for  word.  Editor  will  code) 


21 .       IF  ALL  ANSWERS  TO  CONSUMER  INTERVIEW  WERE  0,  WHY? 

□ 

2  Did  not  choose  to  answer  questions 

1  Not  able  to  answer  questions 

0  Other  


Reviewer's  Impressions  Specific  to  this  Person 


1 .      On  a  scale  of  1  -1 0,  how  good  is  this  person's  life? 


1   =  I  can't  imagine  anything  worse. 

3  =  Poor 

5  =  Mediocre 

7  =  Desirable 

10  =  I'd  switch  lives  in  a  minute. 


24 


RELATIONSHIPS/SOCIAL  CONNECTIONS 


INSTRUCTIONS 

Information  is  gathered  during  the  interview  with  the  person  or  their  representative.  The  person  is  asked  to  identify 
people  who  are  important  to  them  in  the  following  categories.  A  person  can  only  be  listed  in  one  category.  The  reviewer 
should  seek  to  determine  1)  the  frequency  of  contact  during  the  last  year.  Complete  the  analysis  at  the  end  of  the  form 
using  the  information  gathered  from  the  interview.  This  tool  was  adapted  from  the  Social  Network  Analysis  Form 
developed  by  Homer,  Reed,  Ritchie,  Stoner  and  Twain  (1986). 


INDIVIDUAL'S  IN  THE 
PERSON'S  SOCIAL  NETWORK 


How  often  have  you  seen  or  done  things  with 
this  person  in  the  last  year? 


Family 
Members 


Room 
Mates 

Co-Workers 

School 
Mates 


Paid  Staff 
to  provide 
support  or 
services 


Friends 


Neighbors/ 
Others 


ANALYSIS  Of  PERSON'S  SOCIAL  NETWORK  -  EDITOR  ONLY 


Total  Network  See  I   Total  less  Paid People  \     Famly     f  Co-Workers  1  Frlenc 
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INDIVIDUALISM  /  SELF  DETERMINATION 


1 .  Does  this  person  get  up  when  s/he  chooses? 

EL       Yes,  completely  flexible,  determined  by  individual's  needs  and  desires. 

1       Sometimes,  on  weekends  and  holidays,  determined  by  individual's  needs  and  desires. 
0       No,  Person  adheres  to  established  household  routine. 
9  NA 


2.  Does  this  person  choose  when  s/he  eats  her  meals? 

Yes,  completely  flexible,  determined  by  individual's  needs  and  desires. 
1       Sometimes,  on  weekends  and  holidays,  determined  by  individual's  needs  and  desires. 
0       No,  Person  adheres  to  established  household  routine. 
9  NA 


3.  Does  this  person  have  a  choice  about  when  s/he  goes  to  bed? 

2  Yes,  completely  flexible,  determined  by  individual's  needs  and  desires. 

1  Sometimes,  on  weekends  and  holidays,  determined  by  individual's  needs  and  desires. 

0  No,  Person  adheres  to  established  household  routine. 

9  NA 


4.  Is  this  person  involved  in  planning  his/her  meals? 

3       Individual  has  the  major  role  in  the  decision 

2  Staff  encourage  the  person's  participation  to  the  maximum  extent  possible 

1  Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 

0  Staff  do  this  with  little  or  no  input  from  individuall 
9       NA  (No,  individual  not  capable  of  any  input) 

5.  Does  this  person  shop  for  groceries? 

3  Individual  has  the  major  role  in  the  decision 

2  Staff  encourage  the  person's  participation  to  the  maximum  extent  possible 

1  Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 
0  Staff  do  this  with  little  or  no  input  from  individual 

9  NA  (No,  individual  not  capable  of  any  input) 

6.  Does  this  person  shop  for  her/his  own  clothing  and  personal  articles? 

3  Individual  has  the  major  role  in  the  decision 

2  Staff  encourage  the  person's  participation  to  the  maximum  extent  possible 
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1  Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 

0  Staff  do  this  with  little  or  no  input  from  individual 
9  NA  (No,  individual  not  capable  of  any  input) 

7.  Does  this  person  do  her/his  own  banking? 

3  Individual  has  the  major  role  in  the  decision 

2  Staff  encourage  the  person's  participation  to  the  maximum  extent  possible 

1  Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 
0  Staff  do  this  with  little  or  no  input  from  individual 

9  NA  (No,  individual  not  capable  of  any  input) 


8.  Is  this  individual  a  participant  in  deciding  how  household  chores  are  allocated  ? 
3       Individual  has  the  major  role  in  the  decision 

2       Staff  encourage  the  person's  participation  to  the  maximum  extent  possible 
1       Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 
0       Staff  do  this  with  little  or  no  input  from  individuals 
9       NA  (No,  individual  not  capable  of  any  input) 


9.  Does  this  person  participate  in  organizing  social  events,  visits,  parties? 

3  Individual  has  the  major  role  in  the  decision 

2  Staff  encourage  the  person's  participation  to  the  maximum  /extent  possible 

1  Staff  encourage  the  individual's  participation  somewhat  (more  could  be  done) 

0  Staff  do  this  with  little  or  no  input  from  individual 
9  NA  (No,  individual  not  capable  of  any  input) 

1 0.  Did  the  individual  choose  the  place  where  he/she  lives? 

1  Yes,  the  individual  chose  the  place  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  assistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 

0  Other,  not  applicable,  or  unknown  (including  court) 
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11.     Did  the  indvidual  choose  her/his  attendant  and/or  residential  support  staff? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 
0  Other,  not  applicable,  or  unknown 


1 2.  Did  the  individual  choose  his/her  roommates/housemates? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 

0  Other,  not  applicable,  or  unknown 

1 3.  Did  the  individual  choose  his/her  job  or  what  the  individual  does  during  the  week? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 
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1 4.  Does  the  individual  usuallly  choose  what  he/she  does  on  the  weekends  and  evenings? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 

0  Other,  not  applicable,  or  unknown 

15.  Does  the  individual  give  his/her  own  consent  for  medical  care? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 

0  Other,  not  applicable,  or  unknown 

1 6.  Does  the  individual  determine  what  clothes  he/she  wears? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 
0  Other,  not  applicable,  or  unknown 
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1 7.    Does  the  individual  usually  choose  her/his  friends  and  aquaintances? 


1 

2 
3 
4 
5 
6 
7 
8 
9 
0 


Yes,  the  individual  chose  unassisted 

Yes,  with  assistance  from  personal  unpaid  friend 

Yes,  with  asssistance  from  family  member 

Yes,  with  assistance  from  a  legal  guardian 

Yes,  with  assistance  from  agency/school  staff  member 

No,  personal  unpaid  friend 

No,  family  member 

No,  legal  guardian 

No,  agency/school  staff  member 

Other,  not  applicable,  or  unknown 


18.  Does  the  individual  usually  decide  how  her/his  personal  space  (home,  apartment,  bedroom, 
etc.)  is  decorated? 

1  Yes,  the  individual  chose  unassisted 

2  Yes,  with  assistance  from  personal  unpaid  friend 

3  Yes,  with  asssistance  from  family  member 

4  Yes,  with  assistance  from  a  legal  guardian 

5  Yes,  with  assistance  from  agency/school  staff  member 

6  No,  personal  unpaid  friend 

7  No,  family  member 

8  No,  legal  guardian 

9  No,  agency/school  staff  member 
0  Other,  not  applicable,  or  unknown 

1 9.  Does  the  individual  go  to  the  bank  and  deposit  and  withdraw  money? 


1 

Yes,  unassisted 

2 

Yes,  with  assistance  from  personal  unpaid  friend 

3 

Yes,  with  asssistance  from  family  member 

4 

Yes,  with  assistance  from  a  legal  guardian 

5 

Yes,  with  assistance  from  agency/school  staff  member 

6 

No,  personal  unpaid  friend 

7 

No,  family  member 

8 

No,  legal  guardian 

9 

No,  agency/school  staff  member 

0 

Other,  not  applicable,  or  unknown 
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23.     According  to  the  Developmental  Disabilities  Assistance  and  Bill  of  Rights  Act  as  amended  in 
1987,  independence  means  "the  extent  to  which  persons  wirh  developmental  disabilities  exert 
control  and  choice  over  their  own  lives."  On  a  scale  of  1  to  5,  with  1  being  totally  indepenent 
and  5  being  totally  dependent ,  how  independent  do  you  rate  the  individual?  (Circle  your 
response  below) 

J  2  2  4  5 

Totally  independent  Neutral  Not  at  all 

independent 
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I 

1 


HEALTH  AND  SAFETY  ISSUES  INSTRUMENT 


1  2 

3 

4 

Sequence  # 

Editor  SR 

FOR  OFFICE  USE  ONLY 

Site  Code 

EVALUATION  OF  ICFs/MR 

HEALTH  AND  SAFETY  ISSUES 

Developed  by: 

Temple  University 
institute  on  Disabilities/UAP 
September  22, 1995 


Please  Print 

Person's  Social  Security  #  


Person's  Name 


Residential  Provider 


Complete  Site  Mailing  Address 


Primary  Respondent  Name 
)   


Last  First  M. 


City,  Town  State  Zip  Code 


Telephone 


Name  of  Data  Collector 


DEMOGRAPHICS 

1.       TODAY'S  DATE 

/ 


Month  Year 

2.       PERSON'S  DATE  OF  BIRTH 

/  / 


Month  Day  Year 

RESIDENTIAL  PLACEMENT  TYPE 


1  Public  Institution  (Not  ICF/MR) 

2  ICF/MR,  1 6  beds  or  more 

3  ICF/MR,  4  to  15  beds 

4  Community  Living  Arrangement,  group 
home  or  apartment  (30  hours  or  more  of 
staff  time  per  week) 

5  Community  Living  Arrangement,  group 
home  or  apartment  (Less  than  30  hours 
of  staff  time  per  week) 

6  Supported  Living  (30  hours  or  more  of 
staff  time  per  week) 

7  Supported  Living  (Less  than  30  hours  of 
staff  time  per  week) 

8  Home  and  Community  Based  Waiver 
Services  (in-home) 

9  Foster  Family,  Family  Living  Program 

10  With  Family  or  in  own  Home 

1 1  Nursing  Home 

12  Other  (Describe)   


4.       LEVEL  OF  RETARDATION 
(Please  seek  documentation) 


5  Profoundly  retarded 

4  Severely  retarded 

3  Moderately  retarded 

2  Mildly  retarded 

1  Not  mentally  retarded 
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OTHER  DISABILITIES 

Seek  documentation.  Enter  code  in  box. 

Leave  none  blank. 


Blind 


1  Yes 
0  No 

9    Donl  know 


Mental  Disorder 


Autism 


Other  Health  Impairment  (specify): 


Cerebral  Palsy 


Epilepsy,  Controlled 


Epilepsy,  Uncontrolled 


Non-Ambulatory,  Mobile 


Non-Ambulatory,  Non-Mobile 


Speech/Language  Impairment 


Hard  of  Hearing 


Deaf 


Visual  impairment 
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6. 


SEX 


1  Male 

2  Female 


RACE 


1  White 

2  African-American 

3  Latino 

4  Asian 

5  Other 


HEALTH  AND  SAFETY 

MEDICATION: 

1 .      How  many  medications  does  this  individual 
take  on  a  daily  basis?  Count  each  medication 
that  is  given  to  this  person  daily  and  that  is 
prescribed.  Do  not  count  vitamins  or  topical 
ointments.  Enter  the  exact  number. 


□ 


2.     Is  this  individual  receiving  any  medication  for 
the  purpose  of  behavior  change? 


1  Yes 

2  No 


If  this  individual  is  receiving  medication  for 
the  purpose  of  behavior  change,  has  the 
prescribing  physician  certified  the  continuing 
medication  within  the  past  90  days? 

1  Yes 

2  No 

9  NA  -  no  medication  for  behavior  control 
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MEDICAL  NEEDS: 

1 .       In  general,  how  urgent  is  this  person's  need 
for  medical  care? 

3  Generally  has  no  serious  medical  needs 
2    Needs  visiting  nurse  and/or  regular  visits 

to  the  doctor 
1    Has  life-threatening  condition  that 

requires 

very  rapid  access  to  medical  care 
0    Would  not  survive  without  24-hour 
medical  care 


2.  How  many  times  has  this  person  seen  each  of 
the  following  doctors  in  the  past  year? 

  a.  General  Practitioner/Internist 

  b.  Obstetrics/Gynecologist 

  c.  Psychiatrist 

  d.  Dentist 

  e.  Podiatrist 

  f.  All  others 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  problem(s) 

Describe  the  problem: 


Has  there  been  a  problem  getting  medical 
care  during  the  past  year? 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  problem(s) 

Describe  the  problem: 


4.       Has  there  been  a  problem  getting  dental  care 
during  the  past  year? 
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Has  there  been  a  problem  getting  mental 
health  care  during  the  past  year? 


2    No  problem 

1  Minor  or  occasional  problem(s) 
0    Major  or  frequent  problem(s) 


Describe  the  problem: 


.  f .  Fire  requiring  fire  department 

g.  Emergency  program  relocation 

h.  Alleged  misuse  of  client  funds  or  property 
.  i.  Outbreak  of  communicable  disease 

j.  Alleged  violation  of  client  rights 

k.  Other  (please  specify) 


6.       Does  this  person  have  an  Individual  Health 
Care  Plan? 


1  Yes 

2  No 


UNUSUAL  INCIDENT  REPORTS 

1 .         Have  there  been  any  unusual  incidents 

involving  this  individual  in  the  past  6  months 
that  you  have  reviewed? 


1  Yes 


2  No 

2.         If  yes,  how  many?  (Enter  exact  number) 


3.   For  the  unusual  incidents  you  identified,  please 
indicate  how  many  of  each  type  were  reported. 


a. 

Injury/illness  requiring  hospital  admission 

b. 

Serious  illness  requiring  doctor 

c. 

Medication  error 

d. 

Suicide  attempt 

e. 

Unauthorized  absence  -  immediate  jeopardy 

or  more  than  24  hours 

Page -6 


4.       For  the  unusual  incidents  you  identified, 

please  indicate  how  many  of  each  type  were 
reported . 

a.  Rape  (alleged)   Victim 

Rape  (alleged)   Perpetrator 

b.  Assault  (alleged)   Victim 

Assault  (alleged)   Perpetrator 

c.  Abuse  (alleged)  Victim 

Abuse  (alleged)  Perpetrator 

d.  Serious  behavior  incident:  police  involvement 
with  a  client  specific  incident 

 Victim 

 Perpetrator 


Serious  behavior  incident:  emergency 
restraint 


.Victim 

 Perpetrator 


□ 


Are  there  any  instances  of  abuse,  neglect,  or 
mistreatment  that  have  gone  untreated? 
(Please  look  at  1  month's  daily  logs). 

1  Yes 

2  No 

If  yes,  please  describe: 


BEHAVIOR  PLAN 

1 .       Does  this  person  have  a  formal  behavior 
plan? 


□ 


1  Yes 

2  No 
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2. 


Is  there  any  evidence  of  use  of  physical 
restraints? 


1   Yes  7       |f  tne  answer  to  question  6  was  yes,  is  the  use 

of  painful  and/or  noxious  stimuli  part  of  a 
program  plan? 


2  No 

If  yes,  when  did  they  occur? 


□ 


1  Yes 

2  No 
9  NA 


□ 


8.      Are  there  any  unusual  incidents  that  have  not 
been  reported?  (Look  for  documentation) 


If  the  answer  to  question  2  is  yes,  are  the  use  j  |j 

of  restraints  part  of  a  program  plan?  |  |    i  yes 

2  No 

1  Yes  ^  . 

2  No  Describe  the  incident: 

9  NA 


4.       Is  there  any  evidence  of  use  of  Time-Out 
Rooms? 


□ 


1  Yes 

2  No 


If  yes,  when  did  the  use  occur? 

SAFETY 


1. 


□ 


□ 


If  the  answer  to  question  4  is  yes,  is  the  use  of 
the  Time-Out  Room  part  of  a  program  plan? 

1  Yes 

2  No 
9  NA 


Is  there  any  evidence  of  the  use  of  painful 
and/or  noxious  stimuli? 


1  Yes 

2  No 


If  yes,  when  did  the  use  occur? 


□ 


□ 


□ 


Are  there  adequate  number  of  smoke 
detectors  in  this  house/apartment  /living  area 
(one  per  floor  including  attic  and  basement)? 

1  Yes 

2  No 


Do  all  of  the  smoke  detectors  work,  or  does 
the  system  work? 

1  Yes 

2  No 


Are  all  of  the  fire  extinguishers  properly 
charged? 

1  Yes 

2  No 
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4.      Was  there  a  fire  drill  in  the  past  60  days? 


1  Yes 

2  No 

9  NA  (minimal  supervision) 


5.      Was  there  a  fire  drill  in  the  60  days  before 
that? 


1  Yes 

2  No 

9  NA  (minimal  supervision) 


6.       Is  this  person's  living  area  reasonably  clean? 


1  Yes 

2  No 

If  no,  please  explain: 


7.Are  there  any  excessively  unpleasant  odors  in  this 
living  area? 


1  Yes 

2  No 

If  yes,  please  explain: 


8.  Are  there  any  physical  hazards  within  this  living 
area? 


1  Yes 

2  No 


If  yes,  please  explain: 


Page  9 


9.       Does  this  individual  have  all  prescribed 

adaptive  equipment?  (Base  your  judgment  on 
the  individual  plan  as  well  as  any  other 
documentation  such  as  prescriptions,  etc.) 


□  1 


Yes 
2  No 


If  No,  please  describe: 


1 0.       Are  all  living  and  program  areas  accessible  to 
this  individual? 


□ 


1  Yes 

2  No 


1 1 .       Does  this  individual  have  an  Individualized 
Safety  Plan? 


□ 


1  Yes 

2  No 


1 2.       How  safe  is  this  neighborhood? 


□ 


5  Extremely  safe 

4  Safe 

3  Neither  safe  nor  unsafe 

2  Unsafe 

1  Extremely  unsafe 


13. Is  your  neighborhood  accessible  in  a  way  that  meets 
your  needs  (e.g.  ramps,  curb  cuts,  audible 
traffic  signals,  etc.)? 


□ 


3    Completely  accessible 
2    Somewhat  accessible 
1    Not  accessible  at  all 
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DEMOGRAPHICS 

1.       TODAY'S  DATE 

 /  

Month  Year 


2.       PERSON'S  DATE  OF  BIRTH 

 /  /  

Month  Day  Year 


3.       RESIDENTIAL  PLACEMENT  TYPE 


1  Public  Institution  (Not  ICF/MR) 

2  ICF/MR,  16  beds  or  more 

3  ICF/MR,  4  to  15  beds 

4  Community  Living  Anangement,  group 
home  or  apartment  (30  hours  or  more  of 
staff  time  per  week) 

5  Community  Living  Arrangement,  group 
home  or  apartment  (Less  than  30  hours 
of  staff  time  per  week) 

6  Supported  Living  (30  hours  or  more  of 
staff  time  per  week) 

7  Supported  Living  (Less  than  30  hours  of 
staff  time  per  week) 

8  Home  and  Community  Based  Waiver 
Services  (in-home) 

9  Foster  Family,  Family  Living  Program 

10  With  Family  or  in  own  Home 

11  Nursing  Home 

12  Other  (Describe)   


4.       LEVEL  OF  RETARDATION 
(Please  seek  documentation) 


5  Profoundly  retarded 

4  Severely  retarded 

3  Moderately  retarded 

2  Mildly  retarded 

1  Not  mentally  retarded 
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APPENDIX  C 


1       STATE  BY  STATE  ANALYSES  OF  PSYCHOMETRIC 
PROPERTIES  OF  OUTCOME-BASED  QUALITY 
ASSURANCE  INSTRUMENTS 

2       ANALYSIS  OF  ACCREDITATION  COUNCIL  SURVEY 

ANALYSIS  OF  COMMISSION  ON  THE  ACCREDITATION  OF 
REHABILITATION  FACILITIES  SURVEY 
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COLORADO  REPORT 


The  Colorado  site  visit  occurred  September  27-29, 1995.  Individuals  interviewed 
included  Kerry  Stern,  Judy  Ruth,  and  Lynne  Struxness  all  of  the  Colorado 
Division  for  Developmental  Disabilities.  Since  the  site  visit,  members  of  the 
research  team  have  been  in  contact  with  Judy  Ruth  and  Lynne  Struxness,  the 
individuals  most  closely  involved  with  the  Colorado  Progress  Assessment 
Review  (COPAR). 

OVERVIEW 

In  1986,  in  accordance  with  Colorado  state  law,  the  State  Auditor's  Office 
conducted  an  evaluation  of  the  then  Colorado  Division  for  Developmental 
Disabilities  (DDD).  As  part  of  the  review,  the  state  law  suggested  that  the 
progress  of  individuals  should  be  evaluated.  Through  a  contract  with  Conroy  & 
Feinstein  Associates,  the  State  Auditor's  Office  contracted  for  the  development  of 
the  Colorado  Progress  Assessment  Review  (COPAR).  The  COPAR  was 
administered  to  a  sample  of  individuals  receiving  services  through  DDD;  the 
survey  was  re-administered  approximately  one  year  later  to  determine  progress 
during  the  year.  The  State  Auditor's  Office  recommended  to  the  legislature  that 
the  administration  of  COPAR  continue,  and  that  it  be  funded  by  the 
Developmental  Disabilities  Council  (DDC).  The  DDC  funded  the  COPAR  for 
two  years  and  since  that  time  the  COPAR  continues  to  be  funded  by  the 
legislature. 

The  data  collected  include  the  dimensions  of: 

Community  inclusion; 
Consumer  satisfaction; 
Adaptive  skills /independence; 
Behavioral/mental  health  issues; 
Medical  issues; 
Productivity;  and 
Decision-making. 

The  COPAR  assessments  are  conducted  by  an  outside  consulting  firm  that  was 
awarded  a  contract  through  a  Request  for  Proposals  (RFP)  process.  Surveyors 
are  trained  by  Developmental  Disabilities  Services  (DDS)  staff.  The  COPAR  is 
currently  completed  on  a  two-year  cycle,  where  in  the  first  year  the  portions  of 


the  survey  dealing  with  decision-making,  satisfaction  and  activities  are 
completed.  In  Year  two,  data  collection  domains  include:  self-esteem,  friends 
and  relationships,  philosophy  and  activity  levels.  Data  are  collected  annually  to 
a  stratified  random  sample  of  individuals  both  receiving  and  waiting  for 
services 

In  each  year,  a  sample  is  selected  from  each  of  the  21  Community  Centered 
Boards  (CCB),  the  entities  that  contract  for  or  directly  provide  residential, 
vocational  and  case  management  supports  to  individuals.  In  each  CCB  a  sample 
of  10%  or  30  individuals  is  selected,  whichever  is  larger.  Prior  to  data  collection, 
the  DDS  checks  for  representativeness  of  the  sample.  Data  are  then  collected 
and  returned  to  DDS.  The  data  include  interviews  with  individuals  receiving 
and  waiting  for  services,  staff  supporting  individuals  (including  direct  contact 
staff  and  case  managers)  and  parents.  DDS  then  issues  a  statewide  report.  If 
requested,  raw  data  are  made  available  to  each  of  the  CCBs.  In  any  situation 
where  health  and  safety  concerns  are  identified,  they  are  immediately 
communicated  to  DDS  for  follow-up  attention. 

DESCRIPTION  OF  THE  DATA  SYSTEM 

The  COPAR  data  are  currently  maintained  by  Colorado  Developmental 
Disabilities  Services.  The  data  are  analyzed  by  DDS  staff  and  reports  are 
generated  on  a  statewide  basis.  Data  are  analyzed  using  the  Statistical  Package 
for  the  Social  Sciences  (SPSS).  Often  during  the  past  several  years,  specific  areas 
have  been  targeted  for  specialized  reports.  These  reports  have  included  an 
analysis  of  individuals  waiting  for  services  and  a  system  for  screening 
individuals  into  service  need  classifications.  In  addition,  responses  using  the 
COPAR  were  solicited  from  1000  people  in  the  general  population,  facilitating 
comparisons  between  those  1000  individuals  and  individuals  receiving  services 
and  supports  through  DDS. 

RELIABILITY 

Reliability  data  have  not  been  collected  on  the  COPAR  in  many  years.  Surveyor 
staff  receive  training  each  year  that  is  primarily  conducted  by  DDS  staff.  The 
assumption  is  that  the  training  increases  inter-rater  reliability. 

Because  reliability  measures  have  not  been  determined  in  several  years,  the 
Temple  University  Institute  e»  Disabilities  (IOD)  has  developed  a  method  to 
establish  inter-rater  reliability  of  the  instrument.  In  order  to  do  so,  the  COPAR 


was  administered  to  28  individuals  currently  residing  in  residential  settings  in 
Pennsylvania;  10  in  a  large,  public  ICF/MR;  10  in  a  large,  private  ICF/MR;  and  8 
in  a  small,  private  ICF/MR.  Within  one  month,  the  COPAR  was  collected  again 
by  a  different  rater  from  the  IOD,  utilizing  the  same  individuals  as  respondents. 
Training  of  the  reviewers  was  done  by  IOD  staff,  utilizing  training  materials 
developed  by  DDS. 

This  has  been  a  time  consuming  endeavor  requiring  the  approval  of  the 
Pennsylvania  Office  of  Mental  Retardation,  the  two  large  facilities  and  one  non- 
profit agency.  Data  collectors  have  been  trained  to  collect  the  COPAR  data  in 
addition  to  the  Temple  University  Behavior  Development  Survey  in  these 
various  locations.  Raters  have  been  identified  as  Rater  1  and  Rater  2  for  the 
purpose  of  comparing  the  inter-rater  reliability  of  this  instrument.  There  was 
less  than  one  month  between  the  data  collected  by  Rater  1  and  Rater  2  at  the 
same  facility  about  the  same  28  individuals.  Data  collection  was  completed  by 
the  end  of  February,  19%. 

The  results  of  the  reliability  study  were  as  follows: 
Scale  Correlation 

Relationship  Index 
Rights  Index 
Esteem  Index 
Talents  and  Capacities 
Safety  Index 
Decision  Index 

*p=<.01  **p=<.001 

As  the  data  in  the  table  show,  with  the  exception  of  the  Decision  Making  Index, 
the  scales  of  the  COPAR  demonstrate  inter-rater  reliability  within  acceptable 
ranges.  The  decision-making  index  is  a  new  scale  that  was  developed  for  the 
1995  administration  of  the  COPAR.  It  is  clear  that  more  training  is  needed  to 
produce  reliable  information  in  that  area. 


.83** 

.58* 

.84** 

Index  .85** 
.55* 
-.01 


CONCURRENT  VALIDITY 


Developmental  Disabilities  Services  has  not  established  concurrent  validity 
between  the  COPAR  and  other  instruments.  However,  the  Vineland  Scales  of 
Social  Maturity  have  been  collected  for  28  individuals.  Although  these  data 
have  not  been  analyzed  in  relation  to  the  COPAR,  Temple  is  in  possession  of  the 
data  and  has  evaluated  the  feasibility  of  using  the  Vineland  data  to  establish 
concurrent  validity  with  the  COPAR.  However,  because  Temple  University  BDS 
data  were  also  collected,  the  Study  Team  decided  to  proceed  with  the  analysis  of 
those  data  instead.  The  Temple  University  Behavior  Development  Survey 
(BDS)  was  administered  to  28  individuals  at  the  same  time  COPAR  data  were 
collected. 


Once  the  data  were  collected,  scales  were  calculated  on  the  Temple  University 
BDS  and  compared  to  the  scales  calculated  for  the  COPAR.  A  Correlation  matrix 
using  Pearson  Correlation  coefficients  was  produced  to  examine  the 
relationships  between  the  indices  produced  by  the  two  instruments.  The  results 
of  the  analysis  follows. 

COPAR  Scales  Temple  University  BDS  Scales 

Social  Network  Analysis    Dignity        Adaptive  Behavior 


Relationship  Index  .83 
Rights  Index 
Esteem  Index 

Talents  and  Capacities  Index 


.44 


.64** 
.74** 


Consumer  Satis. 
Talents  and  Cap.  Index  .59* 
Esteem  Index  .70** 
Rights  Index 


Individualization  Self-Deter. 


.76" 


A7 


*  p=<  .01  **  p=<.001 

As  the  above  table  shows,  the  COPAR  scales  are  measuring  similar  dimensions 
to  those  of  the  Temple  University  BDS,  indicating  a  strong  degree  of  concurrent 
validity.  As  stated  in  the  reliability  section,  the  one  scale  that  appears  to  be 
problematic  is  the  decision-making  index,  which  is  not  correlated  with  any  of 
the  Temple  University  BDS  scales.  More  work  needs  to  be  done  in  this  area. 


APPLICABILITY 


The  COPAR  instrument  has  been  used  only  in  services  provided  by  or 
contracted  through  the  CCBs.  Therefore  it  has  not  been  used  in  ICF/MR 
facilities  operated  by  the  State  (in  the  Regional  Centers)  or  in  private  ICFs/MR. 
The  data  collected  in  Pennsylvania  has  assessed  the  applicability  of  the  COPAR 
in  settings  serving  individuals  with  the  most  severe  disabilities.  These  data  were 
analyzed  anecdotally  to  determine  the  extent  to  which  the  COPAR  instrument 
can  be  used  in  a  variety  of  settings  and  for  people  with  the  most  severe 
disabilities.  Data  collectors  reported  that  there  were  no  issues  related  to 
collecting  data  for  people  with  the  most  severe  disabilities,  other  than  the  fact 
that  data  collection  using  the  COPAR  is  time  consuming.  When  collecting  these 
data  in  a  large  facility,  the  time  required  to  complete  the  assessments  can  become 
burdensome. 

In  addition,  because  the  COPAR  has  not  been  collected  in  Colorado  in  quite 
awhile,  the  Study  Team  contracted  with  the  same  firm  who  collects  COPAR  data 
in  Colorado  for  some  additional  data  collection.  In  order  to  have  comparative 
outcome  data  for  the  Health  and  Safety  portion  of  the  study,  the  IOD  contracted 
with  the  market  research  firm  to  collect  COPAR  assessments  for  those 
individuals  for  whom  Temple  University  Health  and  Safety  data  had  been 
collected.  This  effort  included  34  individuals,  14  of  whom  were  residents  of  a 
large,  public  ICF/MR.  The  data  indicate  that  those  individuals  experience 
severe  disabilities  including  medical  complexities  and  behavioral  challenges. 

DISCRIMINANT  VALIDITY 

COPAR  data  were  requested  from  the  Colorado  Developmental  Disabilities 
Services.  These  data  have  been  received  and  have  been  analyzed  by  Temple 
staff  to  determine  the  discriminant  validity  of  the  instrument.  A  test-retest  file 
from  1992  and  1993  was  sent  that  included  781  cases  for  analysis.  The  most 
recent  data  (1993)  were  used  for  the  analysis. 

In  order  to  assess  the  discriminant  validity  of  the  instrument,  a  discriminant 
function  analysis  was  performed.  The  purpose  of  discriminant  function  analysis 
is  to  statistically  distinguish  between  two  or  more  groups.  To  distinguish 
between  groups  a  collection  of  discriminating  variables  are  selected,  that 
measure  characteristics  on  which  the  groups  are  expected  to  differ.  Based  on 
discussions  among  the  Study  Team,  it  was  decided  that  the  two  variables  to  be 
used  as  discriminating  variables  would  be  level  of  disability  (mental  retardation) 
and  living  arrangement.  The  extent  to  which  differences  on  the  COPAR  can  be 


explained  by  the  discriminating  variables  was  assessed.  The  outcome  indices 
that  were  examined  for  purposes  of  this  part  of  the  study  included: 

1)  Community  Activity  Index  -  including  items  such  as  whether 
individuals  shopped  for  groceries,  visited  a  museum,  cooked  or  prepared 
meals  in  the  past  two  weeks; 

2)  Self-Esteem  Index  -  including  items  such  as  whether  the  individual  has 
enough  things  to  do  around  the  house,  whether  people  in  the  home  boss 
them  around,  whether  individuals  feel  that  what  they  do  during  the  day 
is  important  to  them  and  to  others; 

3)  Rights  Index  -  including  items  such  as  whether  there  are  places  in  the 
home  that  the  individual  cannot  get  to  due  to  mobility  problems,  whether 
the  individual  can  be  alone  when  they  so  choose,  whether  there  are  rules 
in  the  home; 

4)  Relationship  Index  -  including  items  such  as  whether  the  individuals 
have  friends,  whether  the  individuals  have  romantic  or  intimate 
relationships,  how  well  the  individual  gets  along  with  others;  and 

5)  Decision  Index  -  including  items  such  as  who  chooses  what  the 
individual  eats,  who  chooses  the  decorations  in  the  individual's  room, 
who  decides  how  personal  money  is  spent. 

For  each  of  these  indices,  a  discriminant  function  analysis  was  performed  using 
the  discriminating  variables  of  level  of  mental  retardation  and  living 
arrangement,  to  see  whether  group  membership  (for  both  level  of  mental 
retardation  and  type  of  living  arrangement)  discriminated  scores  on  the  five 
outcome  scales.  For  purposes  of  this  study,  the  living  arrangement  variable  was 
collapsed  from  the  original  16  categories  to  three  categories  which  are: 
individual  setting,  including  own  home,  family  home;  group  setting,  including 
group  home  or  apartment;  and,  institutional  setting,  which  includes  regional 
centers  and  nursing  homes.  For  purposes  of  this  analysis  the  breakdown  of  the 
population  by  type  of  setting  is  as  follows: 


Type  of  Setting  Frequency  Percent 

Individual  Setting  449  61% 

Group  Setting  213  29% 

Institutional  Setting  79  11% 

Unknown  40  Missing 


Total  781 

Level  of  mental  retardation  for  the  group  is  as  follows: 


Level  of  Mental  Retardation 


Frequency 


Percent 


Not  Retarded/Mild 

292 

39% 

Moderate 

181 

24% 

Severe 

144 

19% 

Profound 

126 

17% 

Unknown 

38 

Total 

781 

The  first  set  of  analyses  examined  the  extent  to  which  level  of  mental  retardation 
was  successful  in  discriminating  between  scores  on  the  outcome  measures.  One 
of  the  ways  to  interpret  a  discriminant  function  analysis  is  that  once  a  set  of 
variables  is  found  which  provides  satisfactory  discrimination  for  cases  with 
known  group  memberships,  a  set  of  classification  functions  can  be  derived 
which  will  permit  the  classification  of  new  cases  with  unknown  memberships. 
For  example,  if  the  scores  on  a  given  outcome  variable  are  known,  if  the 
discriminating  variables  work  well,  one  should  be  able  to  determine  from  which 
type  of  living  arrangement  the  individual  came  as  well  as  their  level  of  mental 
retardation  based  on  their  outcome  data.  As  a  check  of  the  adequacy  of  the 
discriminant  functions,  the  original  set  of  cases  can  be  classified  to  see  how 
many  are  correctly  classified  by  the  variables  being  used. 

In  the  first  analyses,  where  the  Study  Team  was  interested  in  discrimination 
based  on  the  Rights  Index,  the  following  results  were  obtained: 


Rights  Index 

Membership 

Actual  Group  Membership 


Predicted  Group 

3  4 


NoMR/Mild  55%  0  27%  18% 

Moderate  46%  0  34%  20% 

Severe  46%  0  24%  29% 

Profound  23%  0  27%  50% 


Percent  of  cases  correctly  classified  =  34.82% 


In  a  variable  such  as  level  of  mental  retardation  where  there  are  four  categories, 
one  would  expect,  by  chance  alone,  to  correctly  classify  25%  of  the  cases.  As 
demonstrated  above,  approximately  35%  of  the  cases  were  correctly  classified, 
indicating  that  the  outcome  score  associated  with  the  rights  index  can  be 
predicted  about  10%  better  than  chance  alone  by  knowing  level  of  mental 
retardation. 

In  the  next  analysis,  a  discriminant  function  analysis  was  run  for  the  rights  index 
with  type  of  living  arrangement 

Rights  Index 

Predicted  Group  Membership 


1  2  3 

Actual  Group  Membership 

Individual  Setting  57%  26%  16% 

Group  Setting  32%  35%  33% 

Institutional  Setting  18%  30%  52% 


Percent  of  cases  correctly  classified  =  50.34% 

In  a  variable  with  three  categories,  such  as  type  of  living  arrangement,  one 
would  predict  correct  classification  of  33%  of  the  cases  by  chance  alone.  With 
the  discriminating  variable  of  living  type,  we  are  able  to  classify  17%  better  than 
what  would  be  expected  by  chance  alone. 

The  next  set  of  analyses  examined  the  Relationship  Index.  The  two  classification 
tables  for  level  of  mental  retardation  and  living  arrangement  are  as  follows: 


Relationship  Index 

Predicted  Group  Membership 


1 

2 

3 

4 

Actual  Group  Membership 

No  MR/Mild 

58% 

0 

17% 

24% 

Moderate 

49% 

0 

19% 

32% 

Severe 

47% 

0 

14% 

39% 

Profound 

32% 

0 

10% 

58% 

Percent  of  cases  correctly  classified  =  35.33% 
Relationship  Index 

Predicted  Group  Membership 


1  2  3 

Actual  Group  Membership 

Individual  Setting  68%  8%  24% 

Group  Setting  53%  6%  41% 

Institutional  Setting  39%  4%  57% 


Percent  of  cases  correctly  classified  =  49.05% 

As  the  tables  show,  for  the  Relationship  Index,  both  of  the  discriminating 
variables  were  able  to  correctly  classify  individuals  somewhat  better  that  what 
would  be  expected  by  chance  alone. 

In  the  next  set  of  analyses,  we  examined  the  Esteem  Index.  The  classification 
tables  for  the  two  discriminating  variables  are  displayed  as  follows: 

Esteem  Index 

Predicted  Group  Membership 
12  3  4 

Actual  Group  Membership 


No  MR/Mild 

61% 

3% 

9% 

27% 

Moderate 

57% 

1% 

10% 

31% 

Severe 

49% 

3% 

7% 

41% 

Profound 

29% 

0% 

0% 

71% 

Percent  of  cases  correctly  classified  =  36.53% 
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Esteem  Index 

Predicted  Group  Membership 


1  2  3 

Actual  Group  Membership 

Individual  Setting  63%  9%  28% 

Group  Setting  52%  9%  38% 

Institutional  Setting  50%  8%  42% 


Percent  of  cases  correctly  classified  =  51.12% 

As  the  tables  show,  for  the  Esteem  Index,  both  of  the  discriminating  variables 
were  able  to  correctly  classify  individuals  somewhat  better  that  what  would  be 
expected  by  chance  alone. 

In  the  next  set  of  analyses,  we  examined  the  Community  Activity  Index.  The 
classification  tables  for  the  two  discriminating  variables  are  displayed  as  follows: 

Community  Activity  Index 

Predicted  Group  Membership 
12  3  4 

Actual  Group  Membership 


No  MR/Mild 

56% 

0% 

10% 

34% 

Moderate 

49% 

1% 

12% 

39% 

Severe 

48% 

0% 

12% 

40% 

Profound 

32% 

1% 

3% 

64% 

Percent  of  cases  correctly  classified  =  35.31% 
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Community  Activity  Index 

Predicted  Group  Membership 


1 

2 

3 

Actual  Group  Membership 

Individual  Setting 

54% 

19% 

27% 

Group  Setting 

49% 

20% 

30% 

Institutional  Setting 

19% 

16% 

65% 

Percent  of  cases  correctly  classified  =  45.41% 

As  the  tables  show,  for  the  Community  Activity  Index,  both  of  the 
discriminating  variables  were  able  to  correctly  classify  individuals  somewhat 
better  that  what  would  be  expected  by  chance  alone. 

In  the  next  set  of  analyses,  we  examined  the  Decision-Making  Index.  The 
classification  tables  for  the  two  discriminating  variables  are  displayed  as  follows: 


Decision-Making  Index 

Predicted  Group  Membership 


1 

2 

3 

4 

Actual  Group  Membership 

No  MR/Mild 

55% 

16% 

17% 

12% 

Moderate 

44% 

14% 

21% 

21% 

Severe 

28% 

15% 

21% 

36% 

Profound 

12% 

4% 

13% 

71% 

Percent  of  cases  correctly  classified  =  41.08% 
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Decision-Making  Index 


Predicted  Group  Membership 


1  2  3 


Actual  Group  Membership 


Individual  Setting 
Group  Setting 
Institutional  Setting 


34% 


34% 


65% 


15% 


14% 


4% 


51% 


62% 


21% 


Percent  of  cases  correctly  classified  =  50.14% 

As  the  tables  show,  for  the  Decision-Making  Index,  both  of  the  discriminating 
variables  were  able  to  correctly  classify  individuals  somewhat  better  that  what 
would  be  expected  by  chance  alone. 

In  summary,  the  data  indicate  that  the  discriminating  variables  are  able  to 
successfully  predict  outcome  scores  on  all  of  the  indices  developed  using  the 
COPAR.  The  ability  to  predict  group  membership  is  somewhat  better  than  one 
would  expect  by  chance  alone.  Philosophically,  this  is  not  what  one  might  wish 
to  observe.  In  other  words,  as  the  field  of  developmental  disabilities  has  moved 
forward,  contemporary  notions  of  outcomes  include  the  belief  that  all 
individuals  regardless  of  where  they  live  and  regardless  of  their  level  of 
disabilities  should  be  afforded  the  opportunity  to  achieve  quality  outcomes. 
These  data  indicate  that  the  achievement  of  outcomes  is,  in  fact,  related  to  where 
an  individual  lives  and  the  level  of  disability  they  experience. 

As  states  continue  to  measure  outcomes  for  all  people  supported,  regardless  of 
where  they  live  or  their  level  of  disability,  we  would  expect  an  increased 
expectation  that  all  individuals  can  be  supported  in  achieving  quality  outcomes 
in  their  lives. 

HEALTH  AND  SAFETY 

As  of  this  date,  Temple  data  collectors  have  collected  health  and  safety 
information  about  35  people  in  Colorado  who  live  in  community  and 
institutional  settings.  This  information  was  compared  to  COPAR  data  collected 
in  these  same  programs  about  the  same  35  individuals.  The  most  recent  COPAR 
data  (collected  by  the  Colorado  DDD)  is  too  old  (in  many  cases  at  least  three 
years)  to  be  valid  for  comparative  purposes.  As  a  result,  new  COPAR  data  need 
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were  collected  on  these  individuals.  This  has  been  an  arduous,  time  consuming 
process  requiring  the  agreement  and  cooperation  of  the  Colorado  DDD,  the 
Community  Centered  Board  (CCB)  responsible  for  the  21  individuals  in  the 
community  and  the  large  public  ICF/MR  responsible  for  the  14  individuals 
living  in  that  facility.  In  order  to  collect  these  data  in  an  expeditious  and  cost 
effective  manner,  Temple  subcontracted  with  the  company  that  has  had  the 
contract  to  collect  the  COPAR  data  in  the  past,  to  collect  these  data.  This 
company  has  been  trained  by  the  Evaluation  Unit  of  the  Division  of 
Developmental  Disabilities  in  Colorado  in  the  administration  of  this  instrument. 


Quality  assurance  information  was  obtained  from  the  Division  of  Developmental 
Disabilities  identifying  any  regulatory  deficiencies  for  the  21  people  in  our  study 
living  in  the  community.  In  addition,  information  on  regulatory  deficiencies 
were  reviewed  from  the  Department  of  Public  Health  for  the  same  21  people. 
For  the  individuals  living  in  the  ICF/MR  facility,  the  OSCAR  3  and  OSCAR  4 
Reports  were  obtained  from  the  Colorado  Department  of  Public  Health.  This 
was  the  only  additional  source  of  data  for  these  individuals. 


All  of  the  above  described  data  have  been  charted  to  compare  with  safety  index 
of  the  COPAR. 

H  S 


AddrtM 

01  02  03  #  MED  MEDND 

•OOCUI 

BP 

SAFESC  SAFE  CDPV  CDF  OA  QA/LS  COP  HS  RBLNDX 

RTSNDX  ESTNOX  TALNDX 

SAFNDX 

DECND 

1.20 

0.37 

15.6 

0.60 

0.31 

0.80 

0.76 

SUE#1 

4   4    1  1.80 

0.33 

23.7 

1.00 

0.33 

2U 

1.00 

38 

28.33 

32.83 

10.17 

2 

13 

SITE  #2 

4   4    1  270 

0.00 

21.3 

0.33 

1.00 

200 

1.00 

51.87 

25.33 

40.33 

13.33 

3 

13.3 

SITE  #3 

4   4   1  200 

0.60 

27.8 

0.60 

0.40 

2.00 

1.00 

37.2 

26 

33 

10 

2 

13 

SITE  #4 

0.00 

0.33 

10.3 

0.33 

0.00 

0.00 

1.00 

0 

0 

1 

1 

30.33 

26 

32.33 

11  67 

2.7 

13 

SITE  #5 

0.00 

1.00 

15.0 

200 

0.00 

0.00 

0  00 

0 

0 

0 

0 

48 

26 

33 

15 

3 

14 

SITE  #6 

0.87 

0.00 

7  3 

0.00 

0.67 

0.00 

1.00 

0 

1 

47  33 

26 

35  33 

14 

3 

15.3 

SITE  #7 

3.00 

1.00 

22.0 

0.00 

0.00 

0.00 

1.00 

0 

0 

0 

0 

48 

28 

40 

13 

3 

17 

SITE  #8 

0.00 

0.33 

0.0 

1.00 

0.00 

0.00 

0.00 

0 

0 

0 

0 

38.67 

26  87 

35 

11.33 

2.3 

13.7 

SITE  #9 

0.00 

000 

10.0 

0.00 

0.00 

0.00 

0.00 

0 

1 

0 

0 

38 

24 

33 

10 

2 

12 

SITE  #10 

0.00 

2.00 

8.0 

4  00 

0.00 

0.00 

0.00 

1 

0 

1 

1 

39 

24 

33 

10 

2 

12 

SITE  #11 

0.00 

1.00 

16.0 

0.00 

0.00 

0.00 

1.00 

0 

0 

0 

0 

37 

28 

34 

10 

2 

13 

SITE  #12 

0.00 

0  00 

0.0 

0.00 

0.67 

0.00 

0.00 

1 

3 

8 

0 

45 

29.67 

33.67 

10 

2 

13 

SITE  #13 

4  00 

1.00 

150 

0.00 

0.00 

000 

1  00 

0 

0 

40 

30 

46 

14 

3 

14 

SITE  #14 

3  00 

0.00 

17.0 

000 

0  00 

0.00 

1.00 

1 

1 

44 

26 

37 

13 

3 

16 

SITE  #15 

0.00 

0.00 

4.0 

0.00 

0.00 

0.00 

1.00 

0 

1 

0 

0 

30 

29 

34 

8 

2 

12 

SITE  #16 

200 

0.00 

13.0 

1.00 

0.00 

0.00 

1.00 

0 

1 

0 

0 

55 

28 

40 

14 

3 

15 

422 

26.97 

34.01 

11.4 

2.4 

13.5 

CONCLUSIONS 


The  COPAR  has  been  used  in  Colorado  for  more  than  nine  years.  As  a  result,  a 
large  data  base  has  been  amassed,  and  the  Division  is  able,  on  an  ongoing  basis 
to  look  at  change  over  time.  Unfortunately,  the  COPAR  is  used  for  systemic 
analysis  and  is  not  considered  to  be  a  part  of  ongoing  quality  assurance  efforts  in 
the  state.  The  COPAR  is  seen  more  as  a  research  protocol,  focusing  on  particular 
areas  of  interest  in  any  given  year.  These  data  could  be  combined  with  the  data 
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collected  through  the  quality  assurance  process  that  includes  the  Health 
Department  surveys  and  Division  monitoring. 

Because  of  the  way  in  which  the  COPAR  data  have  been  used  by  the  Division, 
these  data  are  not  seen  as  particularly  useful  by  provider  agencies,  the 
Community  Centered  Boards,  consumers  or  advocates.  If  resources  were 
available  to  enable  the  Division  to  provide  feedback  to  all  stakeholders,  these 
data  could  be  more  useful  for  planning  at  the  local  level,  and  for  examining 
trends  over  time  at  the  local  level. 

The  COPAR  has  been  demonstrated  to  be  a  statistically  valid  instrument  as 
demonstrated  by  the  reliability  and  validity  studies  conducted  with  the  COPAR 
in  Pennsylvania.  Given  the  strong  statistical  properties  of  the  instrument,  the 
Study  Team  hopes  that  the  COPAR  will  continue  to  be  used  to  measure 
outcomes  for  individuals  and  that  the  data  will  become  more  accessible  to 
families,  individuals  served  and  agencies  supporting  individuals. 
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MASSACHUSETTS  REPORT 


INTRODUCTION 

The  Massachusetts  site  visit  took  place  on  October  5  &  6, 1995  in  Boston,  Massachusetts. 

The  Temple  staff  member  who  participated  in  this  site  visit  was  Robin  Levine.  During 
the  site  visit,  an  introduction  to  the  project's  goals  and  activities  were  described  to  the 
administrative  staff  of  the  Department  of  Mental  Retardation  (DMR).  In  turn,  the  DMR 
staff  described  the  ongoing  development  and  use  of  the  Massachusetts  outcome  based 
quality  assurance  system.  The  following  individuals  were  interviewed  during  the  site 
visit  and  have  been  an  integral  part  of  the  ongoing  project  activities: 

•  Philip  Campbell  -  Commissioner,  DMR 

•  Mary  Cerreto,  DMR,  Assistant  Commissioner,  Office  of  Quality  Enhancement 
(OQE) 

•  June  Rowe  -  DMR,  Assistant  Director  of  OQE 

•  Gail  Grossman  -  DMR,  Assistant  Director  of  OQE 

•  Ronald  Sanfield  -  DMR,  Director  of  Evaluation,  OQE 

•  Laurie  Anderson  -  DMR,  Management  Information  System 

•  Therese  LaCroix  -  DMR,  Evaluation  Unit 

•  Paul  Blake  -  Wrentham  Regional  Center,  Quality  Assurance 

OVERVIEW  -  THE  SURVEY 

The  Quality  Enhancement  Survey  Tool  (QUEST)  is  administered  by  the  Office  of 
Quality  Enhancement  (OQE)  of  the  Department  of  Mental  Retardation.  The  QUEST 
process  incorporates  a  combination  of  Accreditation  Council  and  Title  XIX  certification 
processes.  The  survey  includes  criteria  from  the  Massachusetts  Health  and  Safety 
(H&S)  Licensing  Standards.  The  conceptual  plan  of  the  QUEST  process  was  completed 
in  August  1993.  The  manual  and  instrument  were  completed  in  January  1994. 
QUEST  has  taken  the  place  of  licensing  and  human  rights  monitoring  for  all 
community  programs.  The  agency  is  the  unit  of  certification  based  on  a  35%  sample  of 
individuals.  A  one-third  random  sample  strategy  was  designed  in  order  to  ensure  that 
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all  individuals  served  by  DMR  will  be  surveyed  within  the  first  three  years  of  the  new 
system. 

Six  quality  of  life  (QOL)  areas  are  measured  for  each  individual  in  the  QUEST: 

1.  Rights  and  Dignity 

2.  Individual  Control 

3.  Community  Membership 

4.  Relationships 

5.  Personal  Growth  and  Accomplishments 

6.  Personal  Well-Being 

There  is  also  an  evaluation  of  the  organizational  structure  and  process.  Outcomes  in 
each  quality  of  life  area  are  defined  in  observable  terms  with  suggested  measures  of 
interpretation.  There  are  a  series  of  questions  which  provide  further  clarification  on 
expected  outcome  measures.  Each  outcome  is  rated  and  receives  one  of  the  following 
six  scores: 

1.  exemplary 

2.  exceeds 

3.  achieved 

4.  partially  achieved 

5.  not  achieved 

6.  not  rated 

There  are  four  levels  of  certification  based  on  the  ratings  in  each  of  the  six  quality 
indicators:  two  years  (3  exceeds,  all  other  achieves),  one  year  (all  achieves), 
certification  with  conditions  (any  partials  or  not  achieved)  and  non-certification.  In  the 
event  that  a  team  member  becomes  aware  of  a  situation  in  which  the  individual  is  in 
jeopardy  of  life,  health,  safety  and/or  dignity,  the  surveyor  immediately  reports  this 
situation  to  the  agency  and  the  regional  director  of  DMR. 

QUEST  is  conducted  by  a  survey  team  comprised  generally  of  three  people  but  not 
fewer  than  two  people.   Current  surveyors  are  former  Quality  Assurance,  Human 
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Rights  and  Licensing  Staff.  Other  team  members  may  include  other  DMR  staff,  citizen 
volunteers  or  other  human  service  professionals.  Each  team  member  completes  an 
individual  rating  report  prior  to  a  consensus  meeting.  The  consensus  meeting 
synthesizes  all  the  individual  reports  including  commendations  and  areas  in  need  of 
improvement  for  the  agency.  A  team  leader  is  identified  who  facilitates  the  consensus 
process  and  is  responsible  for  the  administrative  functions  of  the  survey.  Surveyor 
training  is  provided  through  regional  supervisors,  during  the  team  consensus  process 
leading  up  to  the  ratings  and,  through  quarterly  surveyors'  meetings  and  case  based 
learning.  OQE  staff  facilitates  an  intense  look  at  one  person's  issues  (Inquiry  Response 
and  Exploration)  as  an  additional  method  of  training. 

DESCRIPTION  OF  DATA  SYSTEM 

The  data  for  community  residential  sites  are  stored,  analyzed  and  reported  by  the 
Evaluation  Unit  at  the  Central  Office  of  the  Department  of  Mental  Retardation.  Data 
for  each  region  are  entered  at  the  five  regional  offices  and  transmitted  to  the  Central 
Office.  The  first  round  of  data  were  collected  between  July  1994  and  July  1995.  These 
data  were  entered  using  RBase  software.  The  second  round  of  data  (since  July  1995)  is 
being  entered  into  Access  (Microsoft  Relational  Database).  The  database  was  developed 
by  the  MIS  Unit  of  Central  Office. 

There  is  no  cumulative  score  for  the  entire  QUEST  survey.  Each  domain  is  scored  and 
reported  separately.  The  following  key  is  used  for  scoring  each  measure  within  the  six 
Quality  of  Life  domains: 

Exemplary  =  4 
Exceeds  =  3 
Achieved  =  2 
Partially  Achieved  =  1 
Not  Achieved  =  0 
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After  each  measure  is  scored,  a  mean  score  is  determined  for  each  domain.  The 
following  is  a  key  for  the  Quality  of  Life  Outcome  Rating  based  on  mean  scores: 

Exemplary  =  3.5  -  4.0 
Exceeds  =  2.5  -  3.4 
Achieved  =  1.5  -  2.4 
Partially  Achieved  =  .5  - 1.4 
Not  Achieved  =  0  -  .4 

Reliability 

The  inter-rater  reliability  study  of  the  QUEST  was  conducted  in  May  and  June  1994  by 
the  Department  of  Mental  Retardation,  Office  of  Quality  Enhancement  in  the 
Commonwealth  of  Massachusetts.  The  sample  was  comprised  of  30  individuals 
including  six  participants  in  each  of  the  DMR  regions.  The  six  participants  in  each 
region  were  selected  from  different  agencies.  Individuals  receiving  day  support, 
residential  support,  and  both  day  and  residential  support  were  included  in  the  study. 

In  addition  to  determining  inter-rater  reliability,  several  other  objectives  were 
identified  for  this  study.  Additionally,  the  study  was  used  to: 

1.  train  staff  on  specific  measures  in  the  QUEST  instrument; 

2.  isolate  the  outcomes  and  determine  which  measures  require  further 
clarification  to  achieve  better  definition;  and 

3.  to  begin  to  determine  the  applicability  of  the  instrument  in  this  population. 

This  study  was  conducted  with  one  OQE  specialist  designated  as  a  team  member  (Rater 
1).  Rater  1  ratings  were  used  for  the  agency's  certification.  Rater  1  also  participated  in 
the  consensus  process.  Rater  2  was  also  an  OQE  specialist,  but  was  not  a  member  of  the 
team  and  did  not  participate  in  the  consensus  team  process  or  the  determination  of 
agency  certification.  The  purpose  of  Rater  2  was  to  shadow  Rater  1.  Rater  2  evaluated 
the  same  individuals  and  records  as  Rater  1. 

The  scores  of  each  measure  were  compared  for  Raters  1  and  2.  A  computer  analysis 
was  conducted  to  determine  the  reliability  of  the  ratings.  The  reliability  coefficients  for 
all  30  individuals  were  analyzed.  A  difference  of  more  than  1.0,  was  defined  as 
evidence  of  inter-rater  unreliability.  Overall,  the  QUEST  instrument  was  rated  at  72.7 
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%  agreement  This  indicates  that  the  QUEST  instrument  is  reliable.  The  following  is  a 
table  which  displays  the  individual  measures  and  the  percent  agreement  between  Rater 
1  and  Rater  2. 


Measure 

%  Agree 

Home  is  safe 

77% 

I  am  safe  at  home 

77% 

Work  is  safe 

92% 

Free  from  abuse,  neglect  and  mistreatment 

73% 

Health  services  are  responsive 

77% 

Maintain  good  health 

77% 

Economic  resource  meet  needs 

75% 

People  are  respectful 

85% 

I  look  good  to  myself  and  others 

88% 

Life  situation  supports  culture  and  religion 

65% 

Home  reflects  my  personal  taste  and  interest 

73% 

Have  access  in  and  around  home  and  work 

73% 

Home,  work  and  free  time  reflect  value  by  the  community 

69% 

Rights  are  protected 

69% 

Safeguards  for  limitations  in  life 

65% 

Privacy 

81% 

Support  to  exercise  and  safeguard  rights 

65% 

Need  for  fiduciary  is  reviewed  regularly 

69% 

Access  to  personal  information  is  limited 

84% 

Primary  decision  maker 

77% 

Communicate  needs  and  preferences 

85% 

Like  where  live 

73% 

Like  where  work 

88% 

Other  options  for  work 

96% 

Choose  free  time 

88% 

Opinions  are  basis  of  action 

65% 

Participate  in  community  life 

73% 

Part  of  workplace  culture 

92% 

Contribution  to  community 

34% 

Satisfying  and  meaningful  ties 

58% 

Ways  to  meet  other  people 

38% 

Make  and  sustain  friendships 

50% 

Intimacy  is  respected  and  supported 

54% 

Develop  personal  goals 

77% 

Develop  skills  for  self  reliance 

81% 

Working  toward  goals 

77% 

Receive  support  to  achieve  personal  goals 

50% 
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The  two  measures  of  least  agreement  between  Rater  1  and  Rater  2  were  Contribution  to 
Community  (34%)  and  Ways  to  Meet  Other  People  (38%). .  These  data  suggest  that 
further  training  needs  to  be  provided  to  reviewers  on  the  interpretation  and  scoring  of 
these  two  measures. 

Concurrent  Validity 

The  QUEST  instrument  has  not  previously  been  validated  with  other  instruments.  In 
order  to  accomplish  this,  the  QUEST  survey  data  need  to  be  compared  with  data 
collected  by  an  instrument  with  proven  statistical  properties.  The  original  idea  for 
Massachusetts  was  to  bring  trained  QUEST  surveyors  to  Pennsylvania  to  administer  the 
QUEST  to  people  in  the  Philadelphia  area  for  whom  data  had  been  collected  using  a 
statistically  proven  instrument.  Due  to  the  fact  that  the  QUEST  requires  a  team  rating 
and  takes  several  days  to  complete,  the  Study  Team  determined  this  to  be  an  ineffective 
method  in  terms  of  cost  and  time.  Rather,  we  asked  the  Commissioner  of  DMR  to 
approve  the  training  and  hiring  of  a  data  collector  to  collect  the  Temple  University 
Behavior  Development  Survey  (BDS)  in  Massachusetts.  Approval  from  the 
Commissioner  for  this  continued  data  collection  in  Massachusetts  was  received  the  first 
week  of  January  19%.  A  data  collector  was  hired  and  trained  to  collect  these  data. 
Additional  permission  and  consent  was  needed  from  the  individuals  in  the  specific 
community  agencies  in  which  data  would  be  collected.  The  agency  that  agreed  to 
participate  obtained  the  permission  of  each  individual  in  the  study  for  Temple  to  collect 
this  information.  Upon  receiving  the  various  consents,  the  Temple  University  Behavior 
Development  Survey  was  administered  in  February  19%  to  30  individuals  (21  people 
living  in  six  community  residences  and  9  people  living  in  one  ICF/MR)  who  have 
QUEST  data  in  the  Massachusetts  state  system. 

Unfortunately,  because  there  were  only  agencies  surveyed,  producing  two  scores,  the 
amount  of  variance  was  not  sufficient  to  perform  conventional  concurrent  validity 
analyses.  Although  the  TEMPLE  BDS  was  collected  for  30  individuals,  QUEST  scores 
are  produced  for  an  entire  agency;  therefore  all  21  individuals  living  in  community 
settings  received  one  score;  similarly  all  nine  individuals  living  in  the  ICF/MR  received 
the  same  score.  As  an  alternative,  score  on  the  two  sets  of  instruments  were  converted 
to  percentages,  and  areas  of  commonalty  were  compared.  The  following  table  displays 
these  data: 
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MA  Domain 


BDS  Domain 


1.  Rights /Dignity  Dignity 

Site  1  50%  95% 

Site  2  50%  100% 

2.  Individual  Control  Self-Determination 

Site  1  25%  52% 

Site  2  50%  70% 

3.  Community  Membership       Integrated  Activity 

Site  1  0  39% 

Site  2  49%  70% 

4.  Relationships  Social  Network  Analysis 

Site  1  25%  6 

Site  2  27%  17 

5.  Personal  Growth 

Site  1  50% 
Site  2  49% 

6.  Personal  Well-Being 

Site  1  50% 
Site  2  48% 

Again,  because  the  QUEST  items  were  scored  on  a  scale  of  0  through  4,  with  0  meaning 
not  accomplished  and  4  meaning  exemplary,  there  was  not  enough  variance  in  the 
range  of  scores  as  there  was  in  the  TEMPLE  BDS.  Therefore,  the  comparative  data  are 
not  as  meaningful  as  one  might  have  hoped.  As  the  table  above  demonstrates,  for  the 
last  two  QUEST  domains,  personal  growth  and  personal  well-being,  there  were  not 
items  from  the  TEMPLE  BDS  that  were  readily  comparable.  On  the  first  domain,  rights 
and  dignity,  there  was  no  difference  between  the  two  sites  (large  ICF  and  community) 
nor  between  the  data  on  the  TEMPLE  BDS  domain  of  dignity.  However,  QUEST  rated 
the  sites  as  50%  of  the  possible  scale,  while  the  TEMPLE  BDS  rated  the  scale  at  close  to 
the  top  of  the  scale.  For  individual  control,  Site  1  received  a  low  score  and  Site  2  a  mid- 
range  score;  on  the  HCFA  index  the  spread  between  the  two  sites  was  the  same,  but  the 
scores  for  both  sites  were  higher  than  the  QUEST  scores.   In  terms  of  community 
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membership,  again  Site  1  scored  lower  than  Site  2;  a  pattern  similar  to  that  found  with 
the  BDS.  Again,  however,  the  QUEST  scores  on  this  domain  were  significantly  lower 
than  the  TEMPLE  BDS  scores.  Finally,  we  reported  scores  on  the  QUEST  Relationships 
Index  and  compared  them  to  the  TEMPLE  BDS  Social  Network  Analysis.  On  this 
domain,  however,  the  TEMPLE  BDS  scores  could  not  be  converted  to  a  percent  as  the 
raw  data  are  based  on  actual  number  of  relationships.  Therefore,  we  can  only  address 
directional  similarities.  As  the  data  show,  in  Site  1  the  relationships  score  was  slightly 
higher  than  for  Site  2;  the  difference  was  a  bit  greater  on  the  scores  on  the  TEMPLE 
BDS. 

In  summary,  the  concurrent  validity  study  using  the  QUEST  instrument  provided  the 
Study  Team  with  a  conceptual  framework  for  a  more  thorough  study.  However,  given 
the  resources  that  would  have  been  necessary  to  complete  a  thorough  concurrent 
validity  study,  which  would  have  required  visits  to  numerous  sites,  such  a  study  was 
impossible  to  complete  during  this  project.  Hopefully  more  attention  will  be  focused 
on  this  effort  by  the  State  of  Massachusetts  in  the  future. 

Applicability 

The  QUEST  was  administered  at  an  ICF/MR  facility  by  regional  center  staff  as  a  pilot 
to  determine  the  feasibility  of  utilizing  an  outcome-based,  person-centered  quality 
assurance  system  in  large  facilities  where  people  with  severe  and  multiple  disabilities 
live.  In  implementing  the  survey  process  many  staff  began  and  remained  skeptical 
about  the  use  and  value  of  an  outcome-based  system  for  this  group  of  people.  Others 
began  with  caution  and  were  later  able  to  understand  the  value  of  this  new  orientation 
to  people's  preferences,  rights  and  dignity.  As  might  be  expected,  the  ratings  in 
particular  QUEST  domains  that  depend  on  access  to  community  resources  were  low. 
The  rating  for  community  membership  was  "not  achieved"  and  the  ratings  for 
relationships  and  individual  control  were  "partially  achieved".  In  summary,  the 
ICF/MR  pilot  demonstrates  that  the  QUEST  is  applicable  across  a  variety  of  settings. 

The  ICF/MR  Facility  Summary  Report  resulted  in  a  plan  of  correction  based  on  the 
findings.  In  this  way  the  QUEST  was  being  used  as  a  quality  enhancement  tool. 
Similarly,  the  QUEST  is  being  used  as  part  of  the  internal  monitoring  system  in  other 
state  facilities.  Training  of  Qualified  Mental  Retardation  Professionals  in  these 
facilities  is  underway  in  order  to  educate  and  prepare  the  staff  and  programs  for  this 
type  of  survey  process.  QUEST  data  are  stored  at  the  various  regional  centers  and  are 
not  currently  analyzed  by  the  Department  of  Mental  Retardation. 
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Discriminant  Validity 


The  staff  of  the  Evaluation  Unit  of  DMR  has  provided  Temple  staff  with  the  QUEST 
data  collected  between  July  1994  and  June  1995.  These  data  and  data  from  an  ICF/MR 
facility  have  been  analyzed  to  determine  the  instrument's  discriminant  validity  in  areas 
such  as  ability  to  predict  living  arrangements  based  on  individual  ratings. 
Unfortunately,  the  QUEST  data  base  does  not  have  discriminating  variables  such  as 
level  of  disability.  Because  the  QUEST  is  only  used  in  community  settings,  there  is  no 
variability  in  the  variable.  Even  using  the  data  collected  on  the  Temple  BDS,  there 
were  only  two  sites  and  it  would  have  been  virtually  impossible  to  complete  the 
analysis.  Therefore,  it  was  impossible  to  complete  the  discriminant  validity  portion  of 
the  study. 

Health  and  Safety 

In  order  to  demonstrate  the  degree  to  which  the  QUEST  guarantees  and  protects  the 
health  and  safety  of  individuals,  a  comparison  study  was  conducted  based  on  both 
existing  and  new  data.  The  new  data  were  collected  by  a  Temple  data  collector  using 
the  Temple  University  Health  and  Safety  Issues  Instrument.  Information  was  collected 
on  site  for  30  individuals  (21  people  living  in  six  community  residences  and  9  people 
living  in  one  ICF/MR)  who  have  QUEST  Health  and  Safety  data  in  the  Massachusetts 
state  system.  In  addition,  information  was  obtained  from  the  Investigations  Unit  of 
DMR  identifying  any  allegations  of  abuse  or  neglect  for  the  same  30  people  in  our 
study.  The  Investigations  Unit  was  the  only  additional  source  for  information  on 
individuals  residing  in  the  community  because  the  QUEST  has  taken  the  place  of  all 
other  methods  of  monitoring,  licensing  and  certification.  For  the  individuals  living  in 
the  ICF/MR  facility,  the  OSCAR  3  and  OSCAR  4  Reports  were  obtained  from  the 
Massachusetts  Department  of  Public  Health.  This  was  the  only  additional  source  of 
data  for  these  individuals. 

All  of  the  above  described  data  have  been  charted  to  compare  with  Quality  of  Life  Area 
#6  (Person  Weil-Being)  of  the  QUEST.  This  QOL  area  includes  three  measures  of 
quality  which  are  individually  rated  according  to  the  0-4  range  described  in  the 
introduction  to  this  section.  In  order  to  obtain  the  overall  score  for  this  Quality  of  Life 
Area,  the  three  measure  scores  were  added  and  an  average  score  was  determined.  The 
reports  received  from  the  Investigations  Unit  were  reviewed  and  the  number  of  health 
and  safety  issues  were  counted.  This  raw  score  is  recorded  in  the  Investigations 
column.  The  attached  table  displays  the  findings  from  all  the  sources  previously 
described. 
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The  study  team  conducted  non-parametric  correlations  which  produced  the  Spearman 
Correlation  Coefficients  in  order  to  analyze  the  various  health  and  safety  data. 
Significant  correlations  were  found  between  the  Medical  Needs  Scale  of  the  Temple 
University  Health  and  Safety  issues  instrument  and  the  overall  score  of  the  QUEST 
QOL  Area  #6  (described  above).  Additionally,  a  significant  correlation  was  found 
between  the  Behavior  Plan  Scale  of  the  Temple  University  Health  and  Safety  issues 
instrument  and  the  total  score  of  the  QUEST  instrument. 

These  findings  must  be  interpreted  with  extreme  caution  due  to  the  lack  of  variance  in 
the  number  of  cases  analyzed  and  the  small  range  of  actual  scores.  While  data  was 
collected  for  30  individuals,  this  represents  scores  for  only  four  different  agencies  due 
to  QUEST'S  method  of  rating  and  scoring  agencies  not  individuals.  Therefore  the  scope 
of  the  analysis  is  quite  small  with  only  four  cases.  Secondly,  the  actual  scores  do  not 
represent  the  full  range  of  scores.  For  instance,  the  range  of  possible  scores  for  the 
Medical  Needs  Scale  of  the  Temple  University  Health  and  Safety  issues  instrument  is 
from  0-3.  The  actual  scores  reported  ranged  from  0  -  .8.  The  range  for  possible 
QUEST  scores  is  from  0-4.  The  actual  range  for  the  total  QUEST  score  in  this  study 
was  from  1.5-2.  From  these  data,  it  appears  that  health  and  safety  measures  are  not 
necessarily  related  to  outcome  measures  of  the  QUEST. 

Typeof  Keiidence  Ol      03      OS      QUEST      #  INVEST       #  MED      MEDND     fDOC     UI       BP       SAFESC      SAFE     QUEST  HAS 


Community  Provider  #1  N/A  N/A  N/A  1.91  1  1.71  0.00  12.00  057  0.28  2.00  2.00  130 

Community  Provider  #2  N/A  N/A  N/A  1.81  0  0.75  0.00  930  0.00  0.00  0.00  1.00  1.84 

Community  Provider  #3  N/A  N/A  N/A  1.94  0  1.00  0.80  8.70  030  0.60  0.40  130  2.08 

ICF/MR  4  2  1  1.33  0  1.22  036  8.67  0.22  0.00  1.67  1.00  2.00 

TOTAL  1.20  0.43  937  0.30  0.27  1.10  133 

4391 INV.  77  83  0.00  2.00  7.00  0.00  0.00  0.00  0.00  0.00 


Conclusions 


The  Massachusetts  quality  enhancement  system  has  many  strong  components.  It  is 
person-centered  and  based  in  quality  of  life  outcomes  with  observable  and  measurable 
criteria.  It  has  proven  inter-rater  reliability  and  is  applicable  to  settings  serving  people 
with  the  most  significant  disabilities.  More  work  needs  to  be  done  in  the  area  of 
concurrent  validity  to  ensure  thsat  the  QUEST  is  measuring  dimensions  that  are  valid 
and  comparable  to  other  acceptable  instruments  in  the  field. 
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The  QUEST  is  also  utilized  as  a  mechanism  to  provide  technical  assistance  to  providers 
and  to  provide  information  to  the  Operations  Division  of  DMR  for  contracting  with 
providers.  The  survey  team  approach  offers  a  broad  perspective  of  experiences  and  the 
consensus  process  supports  the  proven  reliability  of  the  survey.  The  survey  can  take 
several  days  to  complete  depending  on  the  size  of  the  agency  and  the  number  of 
individuals  to  be  observed.  The  time  factor  can  be  seen  as  both  a  positive  and  negative 
attribute  of  the  QUEST.  Positively,  it  seems  that  a  surveyor  would  have  enough  time  to 
observe  and  gather  information,  thereby  formulating  a  comprehensive  evaluation  of 
the  individual's  quality  of  life.  However,  in  terms  of  replicability,  the  time  and 
resources  needed  may  be  too  much  of  a  commitment  in  this  era  of  downsizing.  This 
quality  enhancement  system  seems  to  include  many  of  the  quality  outcomes  judged  to 
be  important  in  the  lives  of  people  with  developmental  disabilities.  As  the  system  has 
only  been  in  effect  for  two  years,  it  is  not  possible  to  judge  the  long  term  effect  of  the 
QUEST  on  the  overall  quality  of  services  and  supports  to  people  supported  through  the 
Massachusetts  Department  of  Mental  Retardation. 
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MISSOURI  REPORT 


Introduction 

The  Missouri  site  visit  took  place  on  October  17  through  October  19,  1995.  An 
introductory  meeting  was  held  in  Kansas  City  between  project  staff  and  key 
State  and  provider  agency  staff  for  the  purpose  of  being  briefed  on  the  intent  of 
the  HCFA  project  as  well  as  providing  project  staff  with  an  overview  of  the 
quality  assurance  system  for  individuals  with  developmental  disabilities. 
Temple  University's  Institute  on  Disabilities  (IOD)  staff,  Kathy  Miller  met  with 
the  University  of  Missouri  at  Kansas  City's  Institute  on  Human  Development 
staff  to  determine  the  extent  of  the  statistical  testing  applied  to  the  outcome 
measure  survey  instrument.  During  the  site  visit  IOD  staff  also  collected  health 
and  safety  information  on  30  individuals  receiving  residential  support  from  the 
Division  of  Mental  Retardation  /  Developmental  Disabilities  (DMR/DD)  in  the 
St.  Louis  area.  The  following  individuals  were  interviewed  during  the  site  visit 

•  Greg  Kramer,  DMH / DMR/DD,  Director 

•  Kate  McClain,  DMH/ DMR/DD,  Quality  Enhancement  Coordinator 

•  Meg  Hutsler,  DMH/DMR/DD,  Quality  Enhancement  Team  Leader 

•  Donna  Haley,  DMH/DMR/DD,  Quality  Enhancement  Surveyor 

•  Cori  Brown,  University  of  Missouri  at  Kansas  City,  Project  Consultant 

•  Christine  Rinck,  Ph.D.,  University  of  Missouri  at  Kansas  City,  Research 
Director 

Contact  with  key  individuals  in  Missouri  by  IOD  staff  was  continuous 
throughout  the  project  as  systems  for  analysis  of  the  quality  outcome  measures 
were  developed  and  the  need  for  securing  additional  information  was  identified. 

Overview 


Missouri  has  three  quality  assurance  options  for  agencies  providing  supports  to 
people  who  have  developmental  disabilities.  They  are: 
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•  The  Certification  Project,  a  voluntary  quality  enhancement  option  for 
agencies  providing  supported  living  programs  under  the  HCB  Waiver. 
Agency  reviews  under  this  project  began  in  July  of  1995.  This  review  is 
done  every  two  years.  Beginning  in  July  of  1996,  the  Missouri 
Advocates  for  Individuals  with  Developmental  Disabilities  (MOAIDD), 
a  group  of  interested  self-advocates  and  advocates,  will  review  the 
Certification  Project  Agencies  during  the  second  year  of  the  two  year 
cycle. 

•  A  recently  enacted  Missouri  law  allows  an  agency  who  chooses  not  to 
participate  in  the  Certification  Project  to  select  either  the  CARF  or  AC 
accreditation  process.  An  agency  who  attains  either  CARF  or  AC 
accreditation  is  granted  deemed  status  under  the  Certification  Project. 

•  For  those  agencies  providing  non-waiver  residential  services,  e.g., 
ICFs/MR,  traditional  licensure  under  the  auspices  of  the  Department  of 
Mental  Health  is  done  on  an  annual  basis. 

The  Certification  Project  is  a  Quality  Enhancement  model  based  on  the  following 
four  principles:  Community  Membership;  Self  Determination;  Rights;  and 
Meeting  Basic  Needs.  The  instrument  used  to  measure  quality  outcomes  is  based 
on  these  four  principles.  Various  outcomes  are  identified  under  each  principle 
which  are  further  delineated  into  standards.  These  standards  are  scored  on  a 
four  point  scale  by  the  QA  Survey  Team.  The  QA  Survey  Team  is  comprised  of 
the  Team  Leader,  a  Credentialed  Team  Member  and  /or  a  Team  Member  in 
Training,  and  interested  Observers.  The  Team  Leader  is  employed  by  the 
Missouri  Department  of  Mental  Health-Division  of  Mental  Retardation  and 
Developmental  Disabilities.  This  individual  has  ultimate  responsibility  for 
carrying  out  the  on  site  survey  process,  and  reports  directly  to  the  Director  of  the 
DMR/DD.  The  Credentialed  Team  Member  has  completed  all  training  and 
experiential  requirements  and  is  responsible  for  assisting  the  Team  Leader  in 
the  survey  process.  The  Team  Member(s)  in  Training  participate  on  survey 
teams  in  specialized  roles  as  part  of  their  experiential  training.  Observers  are 
those  individuals  who  are  interested  in  experiencing  the  enhancement  process 
but  may  not  be  willing  or  able  to  make  a  commitment  to  the  time  entailed  in 
becoming  a  team  member. 

An  average  score  of  the  standards  is  obtained  for  each  outcome.  The  end  result 
is  an  agency  score  for  each  of  the  outcomes.  Additional  feedback  to  the  Agency 
includes  an  enhancement  plan  which  contains  specific  recommendations, 
timelines,  and  resources.  All  agency  waiver  programs  are  evaluated,  and  a 
sample  of  10%  of  the  consumers  are  interviewed. 


The  consumer/advocate  component  of  the  Certification  Project  overseen  by 
MOAIDD  is  proposing  to  use  an  adapted  version  of  OKAIM  from  Oklahoma 
and  MOVERS  from  the  state  of  Washington  to  conduct  their  quality  outcome 
assessments. 

Description  Of  Data  System 

State  QA  Team  Leaders  are  responsible  for  compiling  and  maintaining  the 
outcome  data  collected  during  an  agency  survey.  Sixteen  Agencies  have  gone 
through  the  Certification  Project  survey  process.  Data  from  these  16  surveys 
were  used  in  our  analysis  of  the  Discriminant  validity  portion  of  our  study  as 
well  as  the  health  and  safety  data  comparison. 

Reliability 

A  pilot  reliability  study,  Medicaid  Waiver  Certification  Standards  A  Pilot 
Reliability  Test,  was  completed  on  the  Certification  Project's  survey  instrument 
in  October,  1995.  The  principal  investigator  of  the  study  was  Christine  Rinck, 
Ph.D.,  of  the  Institute  of  Human  Development,  University  of  Missouri  at  Kansas 
City,  the  State's  University  Affiliated  Program.  There  are  plans  to  conduct 
reliability  studies  on  an  ongoing  basis. 

The  following  section  summarizes  the  analysis  of  the  reliability  study. 

A  four  point  scale  is  used  in  rating  an  agency's  ability  to  meet  the  standards 
under  each  of  the  identified  outcomes  in  the  Certification  Project's  survey.  The 
scale  involves  rating  the  standards  as  a)  passively  prohibitive;  b)  enabling;  c) 
enhancing;  and  d)  optimally  enhancing. 

The  accuracy  of  reliability  was  judged  on  the  criteria  that  the  scores  of  the  two 
surveyors  had  to  be  within  one  point  of  each  other  in  order  to  classify  their 
ratings  as  correct.  For  example,  if  one  surveyor  rated  an  item  1  "passively 
prohibitive"  and  the  other  rated  an  item  2  "enabling"  then  this  answer  was 
considered  to  be  correct.  However,  if  one  rated  an  item  1  "passively  prohibitive" 
while  the  other  rated  an  item  3  "enhancing",  then  the  answer  was  classified  as  a 
mismatch  and  hence,  incorrect 

In  addition,  a  Cronbach's  alpha  score  was  computed  for  each  of  the  scales.  The 
objective  of  this  statistical  measure  is  to  determine  the  extent  to  which  the  items 
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on  a  scale  are  closely  related  to  each  other.  1.0  is  a  perfect  score.  Therefore,  the 
closer  the  score  is  to  1.0,  the  higher  is  the  confidence  with  which  the  findings  can 
be  accepted. 

The  following  is  a  synopsis  of  the  Cronbach's  alpha  score  derived  for  each  scale 
employed  to  measure  quality  outcomes. 

Section  1.0 

Community  Membership 

1.1  Promoting  Acceptance  Through  Community  Involvement:  .97 

1.2  Supporting  and  Promoting  Relationships:  .94 

1.3  Supporting  and  Promoting  Contribution:  .76 

1.4  Facilitating  and  Enhancing  Communication:  .84 

1.5  Facilitating  Community  Involvement  .99 

Section  2.0 
self-determination 

2.1  Promoting  Self-Esteem  Through  Positive  Self-Expression:  .97 

2.2  Maximizing  Individual  Choice  and  Decision  Making:  .60 

2.3  Facilitating  Empowerment:  * 
2.4.  Person  Centered  Planning:  .94 

Section  3.0 
Rights 

3.1  Assuring  Human  Rights,  Dignity,  and  Respect:  .96 

3.2  Assuring  Legal  Rights:  .94 

Section  4.0 
Meeting  Basic  Needs 

4.1  Assuring  and  Promoting  Good  Health:  .94 
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4.2  Assuring  Individual  Safety:  .97 
" Alpha  not  computed 

Conclusion 

Thus,  with  the  exception  of  the  scale  on  maximizing  individual  choice  and 
decision  making,  the  study  indicates  high  reliability. 

Concurrent  Validity 

A  Content  Validity  Study  was  completed  in  July  of  1994.  The  study,  Missouri 
Medicaid  Waiver  Certification  Standards:  A  Validation  Study  was  conducted  by 
Christine  Rinck,  Ph.D.,  Director  of  Research,  Institute  for  Human  Development, 
University  of  Missouri-Kansas  City,  a  University  Affiliated  Program.  This  study 
was  achieved  through  the  following  three  activities.  The  first  activity  brought  a 
focus  group  of  14  persons  with  developmental  disabilities  together  to  discuss  the 
importance  of  the  standards.  A  second  activity  consisted  of  convening  a  group 
of  15  families  with  children /adult  family  members  with  developmental 
disabilities  to  identify  standards  that  were  unclear  and  those  that  did  not  reflect 
the  outcomes.  The  third  activity  identified  six  professionals  throughout  the 
nation  who  have  extensive  experience  in  the  field  of  developmental  disabilities, 
and  asked  them  along  with  the  family  members  who  took  part  in  the  focus 
group  to  rate  the  standards.  This  third  group  rated  each  standard  on  a  five-point 
rating  scale  based  on  its  relationship  to  the  outcomes  and  principles,  and 
identified  the  standards  most  and  least  reflecting  the  outcomes.  This  analysis 
allowed  the  project  to  refine  its  survey  instrument  prior  to  implementing  the 
survey  portion  of  the  Certification  Project. 

No  concurrent  validity  study  of  the  Certification  Project's  survey  instrument  has 
been  done  in  Missouri.  The  IOD  conducted  this  study  to  establish  the  extent  to 
which  the  Certification  Project's  outcome  instrument  and  the  Temple  University 
Behavior  Development  Survey  (BDS)  are  measuring  the  same  underlying 
dimensions.  In  considering  where  to  conduct  the  concurrent  validity 
component  of  the  quantification  studies  we  considered  the  concentrated  effort 
made  by  the  Certification  Project  steering  committee  to  put  their  resources  into 
multi-faceted  values-based  training  during  the  two  years  preceding  the  start  of 
the  agency  surveys.  Survey  team  training  and  experiential  requirements  are 
extensive  and  are  designed  to  be  individualized  according  to  the  needs  of  each 
team  member.  Due  to  the  professional  development  design  of  the  training 
requirements,  it  would  be  too  difficult  to  duplicate  the  Certification  Project 
survey  process  in  Pennsylvania. 
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The  concurrent  validity  study  was  conducted  in  Missouri  by  administering  the 
TEMPLE  BDS  to  a  sample  of  33  individuals  receiving  supports  from  5  agencies 
who  have  gone  through  the  Certification  Project  agency  survey  process.  Staff 
from  the  Institute  for  Human  Development,  UMKC/UAP  and  the 
DMH/DMR/DD  Certification  Project  administered  the  BDS  instruments  in 
Missouri.  Training  on  the  use  of  the  TEMPLE  BDS  was  completed  over  the 
telephone  by  IOD  staff.  Consent  was  obtained  for  the  individuals  for  whom 
information  on  the  TEMPLE  BDS  was  collected. 

Because  there  were  only  5  agencies  included  in  concurrent  validity  data  set, 
many  of  the  correlation  coefficients  were  meaningless.  We  are  instead  doing  a 
more  qualitative  analysis  by  comparing  the  variables  for  each  of  the  sites 
included.  The  data  is  summarized  in  the  following  table. 


1.  COMM.  MEMBERSHIP  INTEG.  ACTIVITY  (PA) 


SITE1 
SITE  2 
SITE  3 
SITE  4 
SITE  5 


42  62 

55  68 

20  63 

11  61 

51  53 


2.  SELF  DETERMINATION 


SELF  DETERMINATION(PA) 


SITE1 
SITE  2 
SITE  3 
SITE  4 
SITE  5 


53 
49 
31 
23 
50 


51 
63 
50 
61 
56 


3.  RIGHTS 


DIGNITY  (PA) 


SITE1 
SITE  2 
SITE  3 
SITE  4 
SITE  5 


54 
58 
56 
43 
78 


100 

100 

98 

100 

97 


4.  MEETING  BASIC  NEEDS 


SITE1 
SITE  2 
SITE  3 
SITE  4 

SITE  5 


62 
100 
48 
51 


71 
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Applicability 


The  Certification  Project  outcome  instrument  was  designed  to  measure 
outcomes  of  supports  and  services  provided  under  the  HCB  waiver  to  persons 
with  developmental  disabilities.  The  survey  portion  of  the  project  is  not  being 
used  at  this  time  in  any  of  Missouri's  six  public  ICFs/MR  facilities  or  16  private 
ICFs/MR  facilities.  We  were  not  able  to  duplicate  the  Certification  Project's 
survey  process  in  Pennsylvania  due  to  the  extensive  training  requirements  of 
the  surveyors.  These  two  factors,  therefore  prevented  us  from  establishing  the 
applicability  properties  of  the  Certification  Project's  survey  instrument  in 
settings  serving  individuals  with  the  most  severe  disabilities. 

Discriminant  Validity 

The  Certification  Project  Agency  Survey  Data  contains  survey  scores  of  16 
agencies.  This  data  is  what  was  used  to  determine  the  extent  to  which  the 
survey  instrument  can  successfully  discriminate  between  individuals  based  on 
their  scores.  An  example  would  be  whether,  based  on  Certification  Project 
Agency  Survey  Data  scores,  residential  settings  can  be  predicted.  In  order  to 
analyze  the  discriminant  validity  of  the  instrument  the  most  recent  Certification 
Project  Agency  Survey  data  have  been  forwarded  to  Temple.  The  data  base 
contained  only  the  Certification  Project  Agency  Survey  Data  and,  since  it  was 
agency  based  contained  no  individuals  characteristics  to  predict.  The  analysis 
have  been  performed  by  combining  the  individual  data  from  the  concurrent 
validity  BDS  data  collected  in  Missouri  with  the  Certification  Project  Agency 
Survey  data.  This  combined  data  file  contained  data  for  33  individuals. 

In  order  to  assess  the  discriminant  validity  of  the  instrument,  a  discriminant 
function  analysis  was  performed.  The  purpose  of  discriminant  function  analysis 
is  to  statistically  distinguish  between  two  or  more  groups.  To  distinguish 
between  groups  a  collection  of  discriminating  variables  are  selected,  that 
measure  characteristics  on  which  the  groups  are  expected  to  differ.  Based  on 
discussions  among  the  Study  Team,  it  was  decided  that  the  two  variables  to  be 
used  as  discriminating  variables  would  be  level  of  disability  and  living 
arrangement.  The  extent  to  which  the  outcome  instrument's  scales  can 
discriminate  between  and  predict  level  of  disability  and  type  of  living 
arrangement  was  assessed.  The  outcome  indices  that  were  examined  for 
purposes  of  this  part  of  the  study  included: 

•  Community  membership; 

•  Self  determination; 
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•  Rights;  and 

•  Meeting  basic  needs. 


For  each  of  these  indices,  a  discriminant  function  analysis  was  performed  using 
the  discriminating  variables  of  level  of  disability  and  type  of  living  arrangement, 
to  see  whether  group  membership  discriminated  scores  on  the  five  outcome 
scales.  For  purposes  of  this  study,  the  living  arrangement  variable  includes: 
group  home /waiver/foster  family,  and  supervised  apartment.  For  purposes  of 
this  analysis  the  breakdown  of  the  sample  by  type  of  setting  is  as  follows: 


Type  of  Setting  Frequency 

Group  Home/Waiver/Foster  7 
Supervised  Living  26 


Percent 

21% 
79% 


Total 


33 


Level  of  disability  for  the  group  is  as  follows: 


Level  of  disability 


Frequency 


Percent 


Mild  18 
Moderate/severe/profound  15 


55% 
45% 


Total 


33 


The  first  set  of  analyses  examined  the  extent  to  which  type  of  living  arrangement 
was  successful  in  discriminating  between  scores  on  the  outcome  measures.  One 
of  the  ways  to  interpret  a  discriminant  function  analysis  is  that  once  a  set  of 
variables  is  found  which  provides  satisfactory  discrimination  for  cases  with 
known  group  memberships,  a  set  of  classification  functions  can  be  derived 
which  will  permit  the  classification  of  new  cases  with  unknown  memberships. 
For  example,  if  the  scores  on  a  given  outcome  variable  are  known,  if  the 
discriminating  variables  work  well,  one  should  be  able  to  determine  from  which 
type  of  living  arrangement  the  individual  came.  As  a  check  of  the  adequacy  of 
the  discriminant  functions,  the  original  set  of  cases  can  be  classified  to  see  how 
many  are  correctly  classified  by  the  variables  being  used.  The  table  below  lists, 
by  outcome  measure,  the  percent  of  cases  correctly  classified  using  type  of 
placement  as  a  discriminating  variable. 
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Percent  of  Cases  Correctly  Classified 
Using  Type  of  Living  Arrangement 


Community  membership  69.70% 

Self  determination  69.70% 

Rights  51.52% 

Meeting  basic  needs  69.70% 


In  a  variable  such  as  living  arrangement  where  there  are  two  possible  categories, 
one  would  expect  to  correctly  predict  group  membership  50%  of  the  time  by 
chance  alone.  As  the  table  above  demonstrates,  on  each  of  the  outcome 
variables,  we  were  able  to  predict  outcome  scores  by  living  arrangement  more 
often  than  would  be  expected  by  chance  alone  on  all  outcome  measures  except 
Rights. 

For  the  first  two  standards,  prediction  of  group  membership  was  20%  better  than 
by  chance  alone.  For  the  third  standard,  Rights,  prediction  of  group 
membership  was  the  same  as  change  alone.  For  the  fourth  standard,  Meeting 
basic  needs,  prediction  of  group  membership  was  20%  better  than  by  chance 
alone. 

The  second  set  of  analyses  examined  the  discriminating  properties  of  the 
variable,  level  of  disability.  The  table  below  lists,  by  outcome  measure,  the 
percent  of  cases  correctly  classified  using  level  of  disability  as  a  discriminating 
variable. 

Percent  of  Cases  Correctly  Classified 
Using  Level  of  Mental  Retardation 


Community  membership  63.64% 

Self  determination  63.64% 

Rights  51.52% 

Meeting  basic  needs  54.55% 


Again,  in  a  variable  with  two  categories,  one  would  expect  correct  classification 
by  chance  alone  in  50%  of  the  cases.  As  the  table  demonstrates,  the  outcome 
scores  correctly  predict  group  membership  by  level  more  often  than  would  be 
expected  by  chance  alone,  for  three  of  the  four  outcome  indices. 
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In  summary,  the  discriminant  validity  of  the  Missouri  outcome  instrument  was  a 
little  better  than  would  be  expected  by  chance  alone,  for  both  type  of  living 
arrangement  and  level  of  disability. 

Health  And  Safety 

Health  and  Safety  data  were  collected  on  30  individuals  residing  in  five  living 
arrangements  in  the  greater  St.  Louis  area.  Four  agencies  were  represented  in 
this  sample. 

Two  of  the  four  agencies  have  gone  through  the  Certification  Project  agency 
survey,  and  the  remaining  two  will  be  completed  in  the  current  fiscal  year.  The 
health  and  safety  information  was  gathered  by  IOD  staff  on  October  18  and  19 
using  the  Temple  University  Health  and  Safety  Issues  Instrument,  developed  by 
the  Temple  University  Institute  on  Disabilities.  The  Missouri  quality  outcome 
instrument  scores  will  be  compared  with  the  data  collected  on  the  health  and 
safety  instrument. 

The  Study  Team  obtained  and  reviewed  the  reports  from  the  Department  of 
Mental  Health's  Bureau  of  Quality  Improvement  on  their  review  of  the  five  sites 
where  data  was  collected  using  the  Temple  University  Health  and  Safety  Issues 
instrument.  The  licensing  surveys  were  done  the  year  before  the  Certification 
Project  was  designated  as  the  certifying  body  for  these  sites.  Our  intent  was  to 
compare  the  results  of  the  Temple  University  Health  and  Safety  Issues  survey 
with  the  licensing  reports.  It  was  not  possible  to  make  any  substantial 
comparisons  between  the  health  and  safety  data  collected  as  part  of  the  study  via 
the  Temple  University  Health  and  Safety  Issues  instrument  and  the  State 
licensing  results.  The  Study  Team  found  that  state  licensing  is  based  on  nine 
value  domains  and  are  reported  in  the  summaries  in  a  narrative  form  that  is  not 
quantifiably  rated.  Therefore,  because  the  licensing  information  is  too 
qualitative  in  form  we  are  unable  to  analyze  this  information  and  compare  it 
with  the  Temple  University  health  and  safety  data. 

The  Certification  Project's  outcome  data  are  reported  by  agency  scores.  A  four 
point  scale  is  used  in  rating  an  agency's  ability  to  meet  the  standards  under  each 
of  the  identified  outcomes  in  the  Certification  Project's  survey.  The  scale 
involves  rating  the  standards  as:  1  =  passively  prohibitive;  2  =  enabling;  3  = 
enhancing;  and  4  =  optimally  enhancing.  An  average  score  is  then  assigned  to 
each  of  the  four  outcome  areas  headings.  As  previously  discussed  in  the 
Overview  Section,  these  four  outcomes  are  Community  Membership,  Self 
Determination,  Rights,  and  Meeting  Basic  Needs. 
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The  average  scores  from  these  four  sections  as  well  as  the  health  and  safety  data 
were  analyzed  to  determine  the  congruence  between  the  data.  The  study  team 
conducted  non-parametric  correlations  which  produced  the  Spearman 
correlation  coefficients  to  analyze  both  sets  of  health  and  safety  data.  Significant 
correlations  were  found  between  the  Neighborhood  Safety  Scale  of  the  Temple 
University  Health  and  Safety  Scale  and  three  of  four  outcomes  from  the 
Certification  Project's  outcome  instrument.  The  three  outcomes  were 
Community  Membership,  Self  Determination,  and  Rights. 

These  findings  must  be  interpreted  with  extreme  caution  due  to  the  lack  of 
variance  in  the  number  of  cases  analyzed,  the  small  range  of  actual  scores,  and 
the  fact  that  for  the  Certification  Project,  scores  are  given  to  facilities,  not 
individuals.  While  data  were  collected  for  30  individuals,  this  represents  scores 
from  only  four  different  agencies  due  to  the  Certification  Project's  method  of 
rating  and  scoring.  Therefore,  the  scope  of  the  analysis  is  limited  with  only  four 
cases. 


PROV 

MED 

MEDND 

DOC 

UI 

BP 

SAFESC 

SAFE 

SI 

S2 

S3 

S4 

17 

0.93 

0.18 

10.1 

0.06 

0 

1.5 

1.6 

18 

0.2 

0.2 

12 

0.20 

0 

1 

0 

5 

2.37 

0.25 

9 

0.87 

0.25 

0 

1 

2.7 

2.8 

3.3 

3 

6 

0 

0 

3 

0.00 

0 

2 

1 

2.6 

2.4 

2.7 
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Conclusions 

Our  foremost  impression  of  the  Certification  Project  is  in  regard  to  the  emphasis 
in  the  design  and  implementation  of  the  project  to  maximize  input  from  all 
levels  of  the  population:  self-advocates,  parents,  interested  citizens,  and  all  levels 
of  the  service /support  delivery  system.  We  view  this  as  a  very  positive  aspect  of 
the  project  insofar  that  the  inclusion  of  a  maximum  number  of  stakeholders 
validates  the  process  by  promoting  acceptance  of  the  project.  Optimally,  this 
will  in  turn  lead  to  acceptance  of  the  principles  on  which  the  outcome  measures 
are  based  which  will  lead  to  real  growth  and  improvement  in  individual's  lives. 

Quantitatively,  the  collaboration  with  the  UAP  is  very  positive,  in  that  the 
instrument,  whose  principles  were  identified  by  consensus,  was  validated  by 
individuals  who  are  experts  in  the  field:  individuals  with  developmental 
disabilities,  family  members  of  children /adults  with  a  developmental  disability 
and  professionals  in  the  field  of  developmental  disabilities.  The  results  from 
the  pilot  reliability  study  were  strong,  and  there  are  plans  in  place  to  continually 
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update  the  reliability  of  the  outcome  instrument.  Unfortunately,  we  are 
examining  the  project  just  as  they  are  getting  started  so  there  is  not  much  data  to 
verify  at  this  time. 
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NEW  YORK  REPORT 


Introduction 

The  New  York  site  visit  took  place  on  September  13  and  September  14, 1995.  An 
introductory  meeting  was  held  in  Albany  between  project  staff  and  key  State 
agency  staff  for  the  purpose  of  informing  state  personnel  of  the  intent  of  the 
HCFA  project  as  well  as  providing  project  staff  with  an  overview  of  the  quality 
assurance  system  for  individuals  with  developmental  disabilities.  Temple 
University's  Institute  on  Disabilities  (IOD)  staff  members,  James  Lemanowicz 
and  Kathy  Miller  met  with  state  staff  to  determine  the  availability  of  a  final 
outcome  measure  survey  instrument  and  the  extent  of  its  use  in  collecting  data. 
During  the  site  visit  IOD  staff  also  collected  health  and  safety  information  on  30 
individuals  receiving  residential  support  from  the  Office  of  Mental 
Retardation/Developmental  Disabilities  (OMR/DD)  in  Albany  and  Rome.  The 
following  individuals  were  interviewed  during  the  site  visit 

•  Thomas  Cuite,  OMR/DD,  Division  of  Quality    Assurance,  Deputy 
Commissioner 

•  Thomas  Articola,  OMR/DD,  Division  of  Quality  Assurance 

•  David  Picker,  OMR/DD,  Division  of  Quality  Assurance 

•  Tom  Richards,  OMR/DD,  Division  of  Quality  Assurance 

Contact  with  the  key  stakeholders  in  New  York  by  IOD  staff  was  continuous  as 
systems  for  analysis  of  the  quality  outcome  measures  were  developed  and  the 
ongoing  need  for  securing  additional  information  was  identified. 

Overview 

The  New  York  Office  of  Mental  Retardation  and  Developmental  Disabilities 
(OMR/DD)  Division  of  Quality  Assurance  (DQA)  developed  a  pilot  model  to 
promote  quality  outcomes  in  supports  and  services  provided  to  persons  with 
developmental  disabilities  known  as  COMPASS.  COMPASS  stands  for: 
Consumerism,  Outcomes,  Management  Plan,  and  Agency  Self  Survey. 
COMPASS  is  a  systems  approach  to  quality  outcomes.  The  system  is  based  on 
the  principles  of  3IP-Independence;  Integration;  Individualization;  and 
Productivity. 
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The  Consumerism  portion  of  COMPASS  is  intended  as  a  means  to  obtain 
consumer  input  in  decisions  affecting  people's  lives.  The  Outcome  element  of 
COMPASS  refers  to  the  actual  tool(s)  created  to  measure  quality  outcomes.  The 
Management  Plan  section  of  the  COMPASS  system  is  envisioned  to  be  a  plan  in 
which  each  participating  agency  would  develop  their  own  unique  blueprint 
addressing  how  their  organization  would  achieve  the  underlying  principles  of 
COMPASS.  The  Self-Survey  element  of  COMPASS  is  intended  to  be  an  agency 
created  system  of  self-monitoring.  This  self-monitoring  was  originally  intended 
to  include  only  regulatory  compliance  and  later  evolved  into  a  means  of 
monitoring  the  other  elements  of  COMPASS,  (i.e.,  consumerism,  management 
plan,  and  outcomes). 

Our  focus  in  the  evaluation  of  COMPASS  is  on  the  Outcome  portion  of  the 
system.  During  the  first  pilot  of  the  COMPASS  system,  three  different  outcome 
instruments  were  developed  by  the  three  service  regions  in  the  State.  Due  to  the 
fact  that  there  was  no  uniform  data  collection,  we  are  unable  to  include  any  of 
this  information  in  our  data  analysis.  A  single  Outcome  Review  Instrument  was 
developed  in  September,  1995.  This  is  the  tool  we  analyzed 

Data  System 

New  York  completed  their  pilot  study  of  the  COMPASS  system  in  September, 
1995,  which  included  the  Outcome  Review.  At  the  end  of  the  pilot  study, 
Outcome  Review  data  was  available  for  9  individuals.  This  is  not  enough  data 
to  be  useful  for  statistical  analysis. 

Reliability 

The  final  Outcome  Review  instrument  was  administered  to  9  individuals  in  New 
York  by  two  individual  raters.  This  number  is  too  small  to  establish  inter-rater 
reliability,  therefore  the  New  York  Outcome  Review  instrument  was 
administered  to  28  individuals  residing  in  Pennsylvania  Intermediate  Care 
Facilities  for  persons  with  Mental  Retardation  (ICFs/MR);  10  individuals 
residing  in  a  large  private  ICF/MR,  10  individuals  residing  in  a  large  public 
ICF/MR  and  8  individuals  residing  in  several  small  ICFs/MR.  The  New  York 
Outcome  Review  instrument  was  administered  to  the  same  28  people  by  two 
separate  data  collectors.  These  two  instruments  were  administered  within  a 
month  of  one  another.  Data  collection  was  completed  at  the  end  of  February, 
19%. 
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The  results  of  the  reliability  study  are  presented  in  the  following  table.  Pearson 
correlation  coefficients  were  calculated  for  each  of  the  scales  on  the  instrument. 


Scale  Correlation 

Independence  -.29 
Integration  -.08 
Individualization  .23 
Productivity  .28 

As  the  table  shows,  none  of  the  correlations  were  significant.  The  data  collected 
indicate  that  the  New  York  instrument  is  not  reliable  when  tested  for  inter  rater 
reliability. 


Concurrent  Validity 

The  Temple  University  Behavior  Development  Survey  Development  Survey 
(BDS)  was  administered  to  the  same  28  individuals  who  were  administered  the 
New  York  Outcome  Review  instrument.  The  Temple  University  BDS  has 
established  properties  of  reliability  and  validity.  The  data  from  both 
instruments  were  collected  and  the  information  obtained  from  the  New  York 
Review  Outcome  instrument  was  compared  with  the  data  gathered  from  the 
BDS  to  determine  concurrent  validity. 

Temple  University  BDS  Scale 
NY  Outcome  Index  Dignity       Self  Determination 

Independence  .45  .43 

Individualization  .42 

As  the  table  demonstrates,  there  are  only  three  areas  of  the  New  York  instrument 
that  were  correlated  with  the  Temple  University  BDS;  however  no  area  reached 
statistical  significance.  The  lack  of  relationships  with  other  scales  such  as 
integration  and  productivity  is  cause  for  concern.  It  is  our  recommendation  that 
New  York  continue  working  on  the  development  of  their  outcome  instrument, 
checking  for  reliability  and  concurrent  validity  as  they  progress  in  development. 
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Applicability 


As  mentioned  during  the  Reliability  section  of  this  report,  the  New  York 
Outcome  Review  instrument  was  administered  in  2  large  Pennsylvania  based 
ICF/MR  facilities,  and  2  smaller  community  based  ICFs/MR.  The  individuals 
residing  in  these  living  arrangements  have  a  wide  range  of  abilities  and  physical 
challenges,  and  the  applicability  of  this  tool  to  persons  with  the  most  severe 
disabilities  was  analyzed  using  this  data.  The  other  problems  with  this 
instrument  need  to  be  addressed  if  New  York  intends  to  use  an  outcome 
assessment  with  individuals  with  a  range  of  disabilities. 

Discriminant  Validity 

As  mentioned  in  the  Reliability  and  Applicability  section  of  this  report,  the  New 
York  Outcome  Review  instrument  was  administered  in  2  large  Pennsylvania 
based  ICF/MR  facilities,  and  2  smaller  community  based  ICFs/MR.  We  have 
data  on  28  New  York  Outcome  Reviews.  In  its  latest  data  collection  for  their 
pilot  study,  New  York  only  had  9  Outcome  Reviews  completed.  Our  original 
intention  was  to  augment  our  28  Outcome  Reviews  with  the  9  New  York 
Outcome  Reviews,  giving  us  37  cases  to  analyze.  However,  the  data  obtained 
from  New  York  has  no  residential  identifiers  which  makes  it  impossible  to 
complete  a  discriminate  between  variables. 

This  portion  of  the  review  determines  the  extent  to  which  the  Outcomes  Survey 
can  successfully  discriminate  between  individuals  based  on  their  scores.  An 
example  would  be  whether,  based  on  Outcomes  Survey  scores,  residential 
settings  can  be  predicted.  In  order  to  analyze  the  discriminant  validity  of  the 
instrument  the  Outcomes  Survey  data  collects  in  Pennsylvania  has  been 
analyzed. 

In  order  to  assess  the  discriminant  validity  of  the  instrument,  a  discriminant 
function  analysis  was  performed.  The  purpose  of  discriminant  function  analysis 
is  to  statistically  distinguish  between  two  or  more  groups.  To  distinguish 
between  groups  a  collection  of  discriminating  variables  are  selected,  that 
measure  characteristics  on  which  the  groups  are  expected  to  differ.  Based  on 
discussions  among  the  Study  Team,  it  was  decided  that  the  two  variables  to  be 
used  as  discriminating  variables  would  be  level  of  mental  retardation  and  living 
arrangement.  The  extent  to  which  differences  on  the  outcome  instrument's 
scales  can  be  explained  by  the  discriminating  variables  was  assessed.  The 
outcome  indices  that  were  examined  for  purposes  of  this  part  of  the  study 
included: 
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•  Independence; 

•  Integration; 

•  Individualization;  and 

•  Productivity. 

For  each  of  these  indices,  a  discriminant  function  analysis  was  performed  using 
the  discriminating  variables  of  level  of  mental  retardation  and  living 
arrangement,  to  see  whether  group  membership  discriminated  scores  on  the  five 
outcome  scales.  For  purposes  of  this  study,  the  living  arrangement  variable 
includes:  group  residence  -15  people,  and  ICF/MR,  16  or  more  people.  For 
purposes  of  this  analysis,  the  breakdown  of  the  sample  by  type  of  setting  is  as 
follows: 

Type  of  Setting  Frequency  Percent 

Group  Residence  4-15  8  29% 

ICF/MR16+  20  71% 

Total  28 

Level  of  mental  retardation  for  the  group  is  as  follows: 

Level  of  Mental  Retardation  Frequency  Percent 

Mild  8  29% 

Moderate  4  14% 

Severe  7  25% 

Profound  9  32% 

TOTAL  28 


The  first  set  of  analyses  examined  the  extent  to  which  type  of  living  arrangement 
was  successful  in  discriminating  between  scores  on  the  outcome  measures.  One 
of  the  ways  to  interpret  a  discriminant  function  analysis  is  that  once  a  set  of 
variables  is  found  which  provides  satisfactory  discrimination  for  cases  with 
known  group  memberships,  a  set  of  classification  functions  can  be  derived 
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which  will  permit  the  classification  of  new  cases  with  unknown  memberships. 
For  example,  if  the  scores  on  a  given  outcome  variable  are  known,  if  the 
discriminating  variables  work  well,  one  should  be  able  to  determine  from  which 
type  of  living  arrangement  the  individual  came  as  well  as  their  level  of  mental 
retardation  based  on  their  outcome  data.  As  a  check  of  the  adequacy  of  the 
discriminant  functions,  the  original  set  of  cases  can  be  classified  to  see  how 
many  are  correctly  classified  by  the  variables  being  used.  The  table  below  lists, 
by  outcome  measure,  the  percent  of  cases  correctly  classified  using  type  of 
placement  as  a  discriminating  variable. 

Percent  of  Cases  Correctly  Classified 
Using  Type  of  Living  Arrangement 


Independence  85.71% 

Integration  46.43% 

Individualization  78.57% 

Productivity  64.00% 


In  a  variable  such  as  living  arrangement  where  there  are  two  possible  categories, 
one  would  expect  to  correctly  predict  group  membership  50%  of  the  time  by 
chance  alone.  As  the  table  above  demonstrates,  on  three  of  the  four  outcome 
variables,  we  were  able  to  predict  outcome  scores  by  living  arrangement 
significantly  more  often  than  would  be  expected  by  chance  alone. 

The  second  set  of  analyses  examined  the  discriminating  properties  of  the 
variable,  level  of  mental  retardation.  The  table  below  lists,  by  outcome  measure, 
the  percent  of  cases  correctly  classified  using  level  of  mental  retardation  as  a 
discriminating  variable. 

Percent  of  Cases  Correctly  Classified 
Using  Level  of  Mental  Retardation 


Independence  35. 71  % 

Integration  21.43% 

Individualization  35.71% 

Productivity  36.00% 


In  a  variable  with  four  categories,  one  would  expect  correct  classification  by 
chance  alone  in  25%  of  the  cases.  As  the  table  demonstrates,  the  outcome  scores 
correctly  predict  group  membership  by  level  of  mental  retardation  more  often 
than  would  be  expected  by  chance  alone,  three  of  the  four  outcome  indices. 
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In  summary,  the  discriminant  validity  of  the  New  York  outcome  instrument  is 
better  than  would  be  expected  by  chance  alone,  for  both  type  of  living 
arrangement  and  level  of  mental  retardation.  Given  the  issues  with  reliability 
and  concurrent  validity,  this  instrument  requires  further  analysis. 

Health  And  Safety 

Health  and  Safety  data  were  collected  using  the  Temple  University  Health  and 
Safety  instrument  on  30  individuals  residing  in  three  living  arrangements  in  the 
Albany/Rome,  N.Y.  area,  by  IOD  staff  on  September  14  and  15,  1995.  Two 
agencies  were  represented  in  this  sample.  Each  of  the  agencies  had  been  part  of 
the  COMPASS  pilot  project.  There  are  data  from  the  New  York  Outcome 
Review  instruments  for  9  people  in  the  entire  state.  These  9  individuals  live  in 
different  regions  throughout  the  state.  None  of  the  individuals  for  whom  the 
Study  Team  collected  data  using  the  Temple  University  Health  and  Safety 
Issues  instrument  in  Albany  and  Rome  were  included  in  this  NY  Outcome 
Review  instrument  sample  of  9  individuals. 

No  individual  outcome  data  were  available  for  any  of  the  individuals  residing 
in  the  Albany/Rome  area  where  the  Study  Team  was  physically  able  to  go  to 
and  collect  data.  The  determination  was  made  during  the  site  visit  to  proceed 
with  collecting  health  and  safety  information.  This  decision  was  based  on  the 
assumption  that  because  COMPASS  is  a  systems  approach  to  attaining  quality 
outcomes  we  could  measure  alternate  indicators  for  these  COMPASS  pilot 
project  sites.  In  the  absence  of  individual  outcome  data,  it  was  determined  that 
comparisons  could  be  made  between  the  agency  ICF/MR  OSCAR  results,  State 
licensing  results  and  the  Temple  University  health  and  safety  data. 

We  made  our  determination  of  how  well  these  three  Compass  sites  protects  the 
basic  health  and  safety  of  the  individuals  it  provides  supports  to  by  comparing 
our  data  to  the  agency's  ICF/MR  survey  reports  and  the  agency's  state  licensing 
reports,  as  displayed  by  the  table.  In  the  analysis  of  the  data  we  conducted  non 
parametric  correlations  and  assessed  the  Spearman  Correlation  Coefficients  and 
found  no  significant  correlations.  This  in  part  is  due  to  the  small  number  of 
cases  in  the  study.  In  addition,  it  is  possible  that  due  to  the  passage  in  time 
since  the  state  collected  data,  and  the  difference  in  the  dimensions  assessed  by 
theTemple  University  BDS  and  the  NY  licensing  and  OSCAR  data,  there  are  no 
relationships  between  the  variables  measured. 
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CONCLUSIONS 


The  design  of  the  NY  pilot  project  did  attempt  to  elicit  interested  party  input  but 
relative  to  other  state  efforts  it  wasn't  as  far  reaching  as  it  could  be.  The  DQA 
surveyors  did  benefit  in  that  they  stated  that  they  felt  more  connected  to  the 
people  they  had  been  surveying  for  all  these  years;  some  reviewers  had  been 
surveyors  for  up  to  20  years. 

Quantitatively,  this  is  a  very  difficult  State  to  analyze  because  there  was  no 
uniform  data  collection  until  the  very  end  of  the  project  at  which  time  data  for 
only  9  people  were  collected.  There  is  uncertainty  at  the  state  level  as  to  the 
future  of  the  COMPASS  project.  From  the  data  collected  during  this  project  for 
28  individuals  using  the  NY  outcome  assessment,  it  is  clear  there  are  problems  in 
terms  of  reliability  and  validity.  If  the  state  decides  to  go  forward  with  the 
development  of  an  instrument,  it  is  critical  that  attention  be  paid  to  establishing 
a  reliable  and  valid  instrument. 
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OKLAHOMA  REPORT 


Introduction 

The  Oklahoma  site  visit  took  place  on  November  7  through  November  9,  1995.  The 
kickoff  meeting  was  held  at  the  State  of  Oklahoma  Developmental  Disabilities  Services 
Division  (DDSD)  in  Oklahoma  City.  Project  staff  from  Temple  University's  Institute  on 
Disabilities  (IOD)  and  from  HSRI  met  first  with  the  Division  Director,  James  Nicholson, 
and  then  with  the  executive  staff.  IOD  and  HSRI  staff  gave  an  overview  of  the  project, 
and  they  presented  an  overview  of  the  Quality  Assurance  system  in  Oklahoma  and 
their  perspective  on  it.  The  Quality  Assurance  system  in  Oklahoma  has  3  parts: 
Performance  Standards  (using  the  AC  standards)  done  by  the  state,  Consumer 
Monitoring  done  by  OK-AIM,  and  the  OSU  Developmental  Disabilities  Quality 
Assurance  Project.  Each  will  be  addressed  separately  below. 

Overview 

Like  many  quality  assurance  systems,  the  Oklahoma  system  originated  from  a  lawsuit 
(Homeward  Bound,  Inc.  v.  .Hissom  Memorial  Center).  The  lawsuit  was  filed  in  1985;  a 
consent  decree  was  issued  in  1990.  Due  to  the  consent  decree,  the  Homeward  Bound 
Review  Panel  and  the  Homeward  Bound  Audit  Team  were  formed.  The  Homeward 
Bound  Audit  Team  does  a  sample  of  50  people  annually.  There  are  1059  Hissom  class 
members,  defined  as  people  who  lived  at  Hissom  Memorial  Center  at  the  time  of  the 
lawsuit.  Hissom  Memorial  Center  closed  in  April  1994. 

The  residential  system  for  people  with  developmental  disabilities  in  Oklahoma  consists 
of  the  following  types: 

Public  institutions  (Enid,  Paul's  Valley,  Greer  Center)  -  575  people 

Private  ICF-MRs  (about  20)  -  2000  people  (no  state  oversight) 

Supported  living  -  700  people 

Group  homes  -  675  people 

Foster  care  - 175  people 

Assisted  living  - 100  people 
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Description  Of  Data  System 


Initial  contact  was  made  with  the  Program  Administrator  of  the  Quality  Assurance 
Unit,  Developmental  Disabilities  Services  Division  who  is  the  contact  person  for  all 
data  requests.  The  AC  Performance  Standards  data  are  available  by  agency.  The 
Oklahoma  AIM  data  are  available  by  site.  The  OSU  QA  Project  data  are  available  by 
individual.  The  Quality  Assurance  Unit  Program  Administrator  was  informed  of  our 
data  needs  for  this  project,  which  are  twofold: 

•  Comparative  data  on  the  30  people  on  whom  Health  &  Safety  data  were 
collected. 

•  A  subset  of  variables  on  their  entire  data  base. 

The  following  is  a  brief  description  of  the  3  parts  of  the  Quality  Assurance  system. 
Performance  Standards  (AC) 

The  contact  person  for  this  part  of  the  Quality  Assurance  System  is  the  Program 
Administrator  of  the  Quality  Assurance  Unit,  Developmental  Disabilities  Services 
Division.  The  data  base  has  state  agency  profiles  of  4500  people  monitored  annually. 
Some  of  the  variables  available  include:  Performance  Score  Percentages,  standards, 
citations,  and  contract  citations.  The  state  AC  survey  team  does  look  at  Health  and 
Safety  issues,  which  may  be  useful  for  comparison  with  the  Health  &  Safety  data 
collected  by  the  Study  Team. 

Consumer  Monitoring  (OK-AIM) 

The  contact  person  for  this  part  of  the  Quality  Assurance  System  is  the  coordinator  of 
the  Oklahoma  AIM  project.  The  project  is  about  3  years  old,  and  monitors  about  500 
people  per  year.  Data  are  site  specific.  This  is  a  community  consumer  monitoring;  data 
are  not  collected  at  the  large  state  facilities.  The  consumer  monitoring  results  in 
monthly  feedback  to  staff.  The  Oklahoma  AIM  data  is  only  now  in  the  process  of  being 
computerized.  For  this  reason,  they  were  not  visited. 

Oklahoma  State  University  Developmental  Disabilities  Quality  Assurance  Project 

The  contact  person  for  this  part  of  the  Quality  Assurance  System  is  Patricia  Bell, 
Department  of  Sociology,  Oklahoma  State  University,  Stillwater,  OK  73125.  Other 
people  interviewed  at  OSUt  A»a*€l»  Fullerton,  Barbara  Murray,  and  Dick  Dodder. 
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The  project  annually  monitors  about  3700  individuals  who  receive  services  from  or 
through  DDSD;  this  includes  all  Hissom  class  members,  all  OBRA  placements,  and 
people  receiving  services  under  the  HCB  Medicaid  Waiver.  The  project  has  been  in 
place  since  1990.  The  assessment  instrument  has  3  parts: 

•  an  interview  with  the  primary  caregiver  for  an  individual:  adaptive  and 
challenging  behavior,  integrative  activities,  medical  needs  and  services,  need 
for  adaptive  equipment,  IHP,  services  and  supports 

•  a  consumer  satisfaction  interview:  satisfaction  with  residential  living 
arrangement,  services,  and  interactions 

•  a  qualitative  site  assessment:  subjective  assessment  of  how  pleasant, 
comfortable,  attractive,  and  safe  the  residential  site  is 

The  Oklahoma  State  University  (OSU)  Developmental  Disabilities  Quality  Assurance 
(DDQA)  Project  was  precipitated  by  the  Homeward  Bound  v.  Hissom  Memorial  Center 
lawsuit.  A  Request  For  Proposals  was  let  by  the  Developmental  Disabilities  Services 
Division  of  the  Department  of  Human  Services.  A  successful  proposal  was  submitted 
by  Oklahoma  State  University.  The  project  brought  in  James  Conroy  and  Celia 
Feinstein,  Quality  Assurance  consultants  from  Pennsylvania,  who  created  the  original 
data  collection  instruments  and  did  the  initial  training  of  data  collectors. 

Since  the  inception  of  the  project,  the  data  collection  instruments  have  been  updated 
annually.  The  "Flag"  form,  which  identifies  important  areas  which  may  require 
immediate  action  by  the  provider  of  service,  has  been  cut  down  to  4  items.  In  the  past 
year,  much  work  has  been  done  by  the  project  to  improve  the  response  rate  of  the 
consumer  satisfaction  interview.  The  original  consumer  interview  data  collection 
instrument  has  been  converted  into  a  picture  book,  which  has  been  undergoing  analysis 
of  its  validity  and  reliability. 

Consequently,  data  collector  training  has  been  expanded  to  a  3-day  training.  It  covers 
the  history  of  the  project,  the  policy  and  procedures  manual,  data  collector  ethics  and 
terms,  the  instruments,  people  first  language,  disability  awareness,  the  use  of  sign 
language,  and  consumer  interview  techniques.  Using  the  resources  available  at  the 
university,  half  of  the  data  collectors  are  graduate  students;  the  other  half  are  human 
service  professionals. 

The  project  has  experienced  difficulty  with  the  Human  Subjects  Review  Board  at  the 
university.  For  this  reason,  they  can  only  give  their  data  to  their  contractor  (State  of 
Oklahoma,  Department  of  Human  Services,  Developmental  Disabilities  Services 
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Division).  All  data  requests  go  to  the  Program  Administrator  of  the  Quality  Assurance 
UnitofDDSD. 

Resources  and  reports  obtained  during  the  visit  to  OSU: 

•  DDQA  Project  Training  Packet  1995-19% 

•  Fullerton,  Douglass,  &  Dodder  (1995).  A  Systematic  Study  Examining 
Reliability  of  Quality  Assurance  Measures. 

•  Conroy  &  Ervin  (1992).  Longitudinal  Assessment  of  Consumer  Outcomes: 
Measuring  Changes  in  Independence,  Productivity,  and  Satisfaction. 

•  Murray  (1994).  Measuring  Changes  in  Consumer  Outcomes:  A  Five  Year 
Longitudinal  Analysis.  1994  Yearly  Report. 

•  Howard,  Fouquet,  &  Dodder  (1994).  Measuring  Consumer  Satisfaction  with  an 
Adaptive  Communication  Device:  A  Test  of  Validity.  Oklahoma  State 
University. 

•  1995  Data  Management  Plan. 
Reliability 

There  have  been  multiple  reliability  studies  done  on  the  OSU  DDQA  Project  data.  A 
preliminary  study  was  conducted  in  Oklahoma  in  1992  to  determine  inter-rater 
reliability  of  the  scaled  items  for  which  interviews  were  conducted  twice  for  a  sub- 
scale  of  86  (i.e.  2%)  DDSD  consumers.  The  range  of  the  correlation  is  from  -1  to  +1.  A 
correlation  coefficient  of  +1  indicates  that  the  scale  scores  were  the  same  each  time  the 
assessment  was  conducted.  Table  1  below  summarizes  the  findings  on  the  correlation 
coefficients. 
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Oklahoma:  Reliability  Correlation  Coefficients  on  Scaled  Items 


Scaled  Item 


Correlation 


Behavior  Development  Scale 
Challenging  Behavior  Severity 
Challenging  Behavior  Frequency 
Consumer  Satisfaction 
Physical  Quality  Scale 
Site  Impression 
Social  Interactions 


.52 


.01 


.05 


.70 


.66 


.93 


.72 


A  coefficient  of  .4  and  below  was  considered  low,  a  score  of  .4  to  .7  was  considered 
moderate,  and  a  score  above  .  7  was  considered  high.  As  shown  in  the  above  table,  the 
findings  were  high  reliability  for  the  Behavior  Development  Scale  (Adaptive  Behavior) 
and  Challenging  Behavior  Severity  scale;  moderate  reliability  for  the  Challenging 
Behavior  Frequency  scale,  Consumer  Satisfaction,  and  the  Social  Interactions;  very  low 
reliability  was  indicated  on  the  scales  on  physical  quality  and  site  impression. 

A  second  study  on  reliability  was  conducted  in  1995  by  the  Department  of  Sociology, 
Oklahoma  State  University,  by  Amanda  Fullerton,  Mark  Douglass,  and  Richard 
Dodder. 

Since  the  beginning  of  the  project,  a  more  in-depth  and  formalized  interviewer  training 
was  put  in  place  by  OSU.  Approximately  half  of  the  interviewers  were  recruited  from 
the  graduate  research  assistants  from  the  Sociology  Department  at  Oklahoma  State 
University.  The  other  half  was  comprised  of  professional  interviewers  who  worked  full 
time  on  this  research  project.  In  both  cases,  the  interviewers  had  worked  on  this  case  for 
an  average  length  of  two  years  with  less  than  half  new  interviewers  each  year. 

The  training  process  for  interviewers  was  as  follows: 

•  The  history  and  objective  of  the  project  were  detailed  at  the  workshop. 

•  The  population  being  studied  was  specified  in  a  "Disability  Awareness"  section 
in  training  materials. 

•  The  interviewers  were  trained  on  sign  language  for  the  "Consumer  Satisfaction" 
section  of  the  instrument  along  with  basic  interviewing  skills. 
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•  A  Survey  Instruction  Manual  and  a  Codebook  were  distributed  among  the 
project  staff. 

•  The  directors  of  the  project  staff  with  the  aid  of  experienced  interviewers 
trained  new  interviewers  on  the  meaning  of  each  item,  ways  of  interpreting  the 
responses,  and  the  entering  of  responses  that  can  be  computer  scanable  as 
documented  by  the  Codebook. 

•  The  new  interviewers  engaged  in  role-playing  with  experienced  interviewers  to 
highlight  situations  which  would  lead  to  incorrectly  interpreted  responses. 

•  A  brief  summary  was  presented  to  explain  the  data  analyses  process.  The 
utilization  of  the  data  was  explained  as  well. 

The  third-day  training  was  conducted  by  speech  pathology  consultants  who  detailed 
the  specific  characteristics  of  this  population  with  developmental  disabilities  and  the 
interviewing  techniques  specific  to  this  population.  The  interviewers  were  also  trained 
on  the  Adaptive  Communication  Device  (ACD)  which  is  a  tool  developed  by  the 
project  staff  to  facilitate  communication  during  the  Consumer  Satisfaction  section  of  the 
instrument. 

New  OSU  interviewers  were  paired  with  experienced  interviewers  for  a  two  week  in- 
field training  to  observe  interviewing  styles  in  different  types  of  placements.  The 
experienced  interviewers  supervised  the  first  in-field  interviews  conducted  by  the  new 
interviewers.  Finally,  the  Senior  Research  Analyst  observed  each  interviewer  before 
they  began  their  interviewing  without  supervision. 

This  study  was  a  four  independent  variable  cross-classification.  The  variables  are:  a) 
verbal/nonverbal;  b)  test-retest/inter-rater;  c)  parent/other  caregiver;  d)  mild  to 
moderate/severe  to  profound  retardation.  There  were  112  individuals  in  the  sample 
with  16  cells  with  7  subjects  in  each  cell. 

The  scheduling  secretary  on  the  Developmental  Disabilities  Quality  Assurance  Project 
at  Oklahoma  State  University  designated  each  interviewer  as  the  inter-rater  (different 
interviewer)  or  test-retest  (same  interviewer).  Based  on  scheduling  needs,  verbal  ability 
and  geographic  region  the  interviewers  were  assigned  to  each  other.  Two  interviews 
had  to  be  completed  within  a  14-day  period  for  each  individual  in  the  sample. 

The  project  personnel  contacted  the  sample  population  to  ensure  that  the  interviewers 
completed  the  assessment  in  person  and  to  ensure  that  the  interviewers  adhered  to  the 
outlined  procedures. 


6 


Overall,  the  study  indicated  high  reliability.  Scales  measuring  Adaptive  Behavior, 
Social  Integration,  Productivity,  and  Consumer  Interview  all  had  reliability  coefficients 
greater  than  .80.  For  the  scales  on  challenging  behavior,  frequency  and  severity,  a 
Pearson  R  of  .73  and  .70  were  recorded,  respectively. 

The  1995  reliability  coefficients  for  scales  in  the  instrument  were  as  follows: 


Demographics 

Sex  .98 

Race  1.0 

Date  of  Birth  .96 

Class  Status  .87 

Functioning  Level  .79 
Scales 

Adaptive  behavior  .99 
Challenging  behavior  (frequency)  .73 

Challenging  behavior  (severity)  .70 
Consumer  interview  .83 

Social  Integration  .89 

Productivity  .85 


In  1995,  reliability  was  done  on  individual  data  only. 

In  summary,  based  on  the  1992  and  1995  reliability  studies  done  in  Oklahoma,  the 
scales  used  for  the  OSU  Quality  Assurance  Project  are  highly  reliable.  The  reliability 
tests  done  already  by  OSU  are  sufficient.  There  is  no  need  for  further  reliability  testing 
of  the  Oklahoma  data. 

Concurrent  Validity 

The  Adaptive  Behavior,  Challenging  Behavior,  Consumer  Interview,  and  Social 
Integration  scales  used  in  Oklahoma  are  identical  to  the  scales  currently  used  in 
Pennsylvania.  A  concurrent  validity  analysis  would  be  redundant.  For  this  reason,  a 
concurrent  validity  analysis  does  not  need  to  be  done. 

Applicability 

The  OSU  Quality  Assurance  system  has  been  used  in  a  variety  of  settings.  The  project 
annually  collects  data  in  sites  ranging  from  Assisted  Living  settings  to  HCB  Waiver 
settings  to  Group  Homes  to  Public  Institutions.  They  collect  data  on  individuals  who 
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have  a  wide  range  of  disabilities.  For  this  reason,  it  is  obvious  that  the  QA  system  is 
applicable  to  a  variety  of  settings. 

Discriminant  Validity 

OSU  Longitudinal  Assessment  data  were  requested  from  the  Quality  Assurance  Unit  of 
DDSD.  Through  a  consultant  working  on  the  project  a  random  sample  of  300 
individuals  was  selected  and  forwarded  to  Temple  University  for  analysis.  These  data 
have  been  received  and  have  been  analyzed  by  Temple  staff  to  determine  the 
discriminant  validity  of  the  instrument  The  data  were  collected  in  1995. 

In  order  to  assess  the  discriminant  validity  of  the  instrument,  a  discriminant  function 
analysis  was  performed.  The  purpose  of  discriminant  function  analysis  is  to 
statistically  distinguish  between  two  or  more  groups.  To  distinguish  between  groups  a 
collection  of  discriminating  variables  are  selected,  that  measure  characteristics  on 
which  the  groups  are  expected  to  differ.  Based  on  discussions  among  the  Study  Team, 
it  was  decided  that  the  two  variables  to  be  used  as  discriminating  variables  would  be 
level  of  disability  (mental  retardation)  and  living  arrangement.  The  extent  to  which 
differences  on  the  OSU  instrument's  scales  can  be  explained  by  the  discriminating 
variables  was  assessed.  The  outcome  indices  that  were  examined  for  purposes  of  this 
part  of  the  study  included: 

•  Number  of  outside  events  during  the  past  week  (inclusion); 

•  Adaptive  Behavior; 

•  Consumer  Interview; 

•  Challenging  Behavior  Frequency; 

•  Challenging  Behavior  Severity;  and 

•  Number  of  hours  of  work  or  education  per  month. 

For  each  of  these  indices,  a  discriminant  function  analysis  was  performed  using  the 
discriminating  variables  of  level  of  mental  retardation  and  living  arrangement,  to  see 
whether  group  membership  (for  both  level  of  mental  retardation  and  type  of  living 
arrangement)  discriminated  scores  on  the  five  outcome  scales.  For  purposes  of  this 
study,  the  living  arrangement  variable  was  collapsed  from  the  original  18  categories  to 
three  categories  which  are:  group  home,  including  group  home  4-6  people  and  group 
home  more  than  7  people;  foster/individual  setting,  foster  care,  independent  living 
and  adult  companion;  and,  assisted  or  supported  living.  Interestingly,  in  the  random 
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sample  selection,  no  one  living  in  congregate  care  settings  was  selected.  For  purposes 
of  this  analysis  the  breakdown  of  the  sample  by  type  of  setting  is  as  follows: 


Type  of  Setting  Frequency  Percent 

Group  Home  138  46% 

Foster/Individual  Setting  40  13% 
Assisted  or  Supported  Living              122  40% 


Total  300 

Level  of  mental  retardation  for  the  group  is  as  follows: 

Level  of  Mental  Retardation  Frequency  Percent 


Not  Retarded/Mild  121  40% 

Moderate  69  23% 

Severe  41  14% 

Profound  53  18% 

Unknown  16  5% 


Total  300 

The  first  set  of  analyses  examined  the  extent  to  which  type  of  living  arrangement  was 
successful  in  discriminating  between  scores  on  the  outcome  measures.  One  of  the  ways 
to  interpret  a  discriminant  function  analysis  is  that  once  a  set  of  variables  is  found 
which  provides  satisfactory  discrimination  for  cases  with  known  group  memberships,  a 
set  of  classification  functions  can  be  derived  which  will  permit  the  classification  of  new 
cases  with  unknown  memberships.  For  example,  if  the  scores  on  a  given  outcome 
variable  are  known,  if  the  discriminating  variables  work  well,  one  should  be  able  to 
determine  from  which  type  of  living  arrangement  the  individual  came  as  well  as  their 
level  of  mental  retardation  based  on  their  outcome  data.  As  a  check  of  the  adequacy  of 
the  discriminant  functions,  the  original  set  of  cases  can  be  classified  to  see  how  many 
are  correctly  classified  by  the  variables  being  used.  The  table  below  lists,  by  outcome 
measure,  the  percent  of  cases  correctly  classified  using  type  of  placement  as  a 
discriminating  variable. 

Percent  of  Cases  Correctly  Classified 
Using  Type  of  Living  Arrangement 


9 


#  of  Events  Outside  the  Home  33.45% 

During  the  Past  Week 

Adaptive  Behavior  45.42% 

Consumer  Satisfaction  27.94% 

Challenging  Behavior  Frequency  44.67% 

Challenging  Behavior  Severity  44.00% 

Hours  of  Work/Education/Month  40.16% 


In  a  variable  such  as  living  arrangement  where  there  are  three  possible  categories,  one 
would  expect  to  correctly  predict  group  membership  33%  of  the  time  by  chance  alone. 
As  the  table  above  demonstrates,  on  each  of  the  outcome  variables  with  the  exception  of 
consumer  satisfaction,  we  were  able  to  predict  outcome  scores  by  living  arrangement 
slightly  more  often  than  would  be  expected  by  chance  alone.  One  of  the  problems  with 
the  sample  selected,  is  that  there  is  less  variance  in  terms  of  living  arrangement  than  in 
the  population,  as  is  demonstrated  by  the  fact  that  no  one  living  in  congregate  care 
settings  was  included  in  the  sample. 

The  second  set  of  analyses  examined  the  discriminating  properties  of  the  variable,  level 
of  mental  retardation.  The  table  below  lists,  by  outcome  measure,  the  percent  of  cases 
correctly  classified  using  level  of  mental  retardation  as  a  discriminating  variable. 

Percent  of  Cases  Correctly  Classified 
Using  Level  of  Mental  Retardation 


#  of  Events  Outside  the  Home  33.45% 

During  the  Past  Week 

Adaptive  Behavior  64.18% 

Consumer  Satisfaction  39.38% 

Challenging  Behavior  Frequency  26.06% 

Challenging  Behavior  Severity  24.65% 

Hours  of  Work/Education/Month  18.30% 


In  a  variable  with  four  categories,  one  would  expect  correct  classification  by  chance 
alone  in  25%  of  the  cases.  As  the  table  demonstrates,  and  as  one  might  expect,  with 
level  of  mental  retardation  as  the  discriminating  variable,  adaptive  behavior  is  correctly 
classified  64%  of  the  time.  Because  of  the  correlation  between  the  two  measures,  this  is 
not  surprising.  For  social  integration  and  consumer  satisfaction,  classification  is 
somewhat  better  than  by  chance  alone.  Challenging  behavior  frequency  and  severity 
and  hours  of  work/ education  are  predicted  no  better  than  by  chance  alone.  Again, 
because  the  type  of  living  arrangement  in  the  sample  is  restricted  to  community  living, 
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there  is  less  variation  in  the  group;  people  with  the  most  severe  disabilities  are 
underrepresented  in  the  sample. 

In  summary,  the  discriminant  validity  of  the  OSU  instrument  is  better  than  would  be 
expected  by  chance  alone,  for  both  type  of  living  arrangement  and  level  of  mental 
retardation. 

Health  &  Safety 

Part  of  the  reason  for  the  site  visit  to  Oklahoma  was  to  collect  Health  &  Safety 
assessments.  Health  and  Safety  data  were  collected  for  30  people.  Residential 
placement  types  visited  included:  a  public  institution,  a  private  ICF-MR,  and  several 
group  homes.  The  purpose  of  this  data  collection  was  to  examine  the  congruence 
between  the  health  and  safety  data  collected,  the  Oklahoma  outcome  measures,  and 
other  measures  of  health  and  safety.  The  other  measures  of  health  and  safety  collected 
in  Oklahoma  include  the  Quality  Assurance  Performance  Surveys  and  the  number  of 
contract  deficiencies  using  the  DDSD  contract  requirements.  DDSD  QA  staff  conduct 
annual  performance  surveys  in  all  ICFs/MR  and  in  community  residential 
arrangements.  If  a  provider  reaches  75%  compliance  with  the  standards  they  receive  a 
certificate  of  achievement;  85%,  they  receive  a  certificate  of  excellence  and  a  waiver  of 
the  next  year's  review.  In  terms  of  contract  deficiencies,  the  number  of  deficiencies  is 
reported  by  agency.  In  both  of  these  efforts,  agency,  rather  than  individual  scores  are 
obtained.  A  table  of  the  health  and  safety  data  by  site  is  included. 

These  scores  as  well  as  theTemple  University  Health  and  Safety  scores  and  the  OSU 
Longitudinal  Assessment  scores  were  entered  onto  a  mainframe  computer  and  a 
Spearman  correlation  was  performed  to  examine  the  relationships  between  the 
variables.  Because  many  of  the  indices  are  site  specific  (QA  Performance  Surveys  and 
contract  deficiencies)  the  analyses  were  restricted  to  only  four  sites  (the  public  ICF/MR, 
private  ICF/MR  and  HCB-funded  group  homes).  As  a  result,  there  were  no  significant 
correlations  found  between  the  outcome  data  and  the  various  other  health  and  safety 
and  quality  assurance  mechanisms. 
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Addr««l  01  02  03  ACP       CCNT  t  MED  MEDND  *D0C  UI  BP 

0.53     0.30  19.4  0.60  0.00 

SITE  »1  89.3      12    0.83    0.00  16.0  0.17  0.00 

SITE  12  94      19    0.33     0.33  32.0  0.00  0.00 

SITE  #3  85.2      70    0.00    0.25  18.7  0.00  0.00 

SITE  #4  91        8     1.50    0.67  11.5  2.80  0.00 


SAFESC  SAFE  QAMED  IDOC  SI  AB  CBF  CBS  CSAT 

0.00  0.20  2.75    3.2  5.2  70.8  92.2  94.6  85.2 

0.0O  0.00  3.00    2.8  4.8  76.4  93.7  9S.0  83.0 

0.00  1.00  2.70    4.7  6.2  90.2  96.9  99.3  88.6 

0.00  0.00  2.70    2.4  5.0  32.9  89.3  93.0  78.4 

0.00  0.00  2.75    5.0  5.7  87.2  95.3  95.3  96.0 


Conclusions 


Overall,  the  Quality  Assurance  system  developed  by  DDSD  is  very  impressive.  Data 
collection  has  been  done  on  a  large  number  of  people  (3700  per  year)  over  a  long  period 
of  time  (5  years)  in  a  variety  of  residential  settings.  Change  over  time  can  be  measured 
on  a  statewide  basis. 


The  outcome  measures  used  in  Oklahoma  are  reliable,  based  on  several  studies 
performed  by  Oklahoma  State  University.  The  scales  used  in  Oklahoma  were  not 
analyzed  for  concurrent  validity;  they  are  identical  to  the  scales  used  in  Pennsylvania 
for  comparison.  The  Quality  Assurance  system  is  used  in  residential  settings  ranging 
from  public  institutions  to  assisted  living  sites,  and  includes  individuals  ranging  from 
people  with  mild  disabilities  to  people  with  severe,  multiple  disabilities.  The  data  are 
used  to  improve  services  to  individuals.  The  discriminant  validity  demonstrates  that 
outcomes  can  be  predicted  better  than  would  be  expected  by  chance  alone  when  using 
the  discriminating  variables  of  type  of  living  arrangement  and  level  of  mental 
retardation.  Health  and  Safety  data  appear  not  to  be  correlated  with  the  outcome  data, 
however,  this  should  be  interpreted  with  caution,  as  the  number  of  observations  is 
small  and  the  variance  across  variables  is  small. 
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OREGON  REPORT 


The  Oregon  site  visit  was  conducted  on  October  24-26,  1995.  Individuals 
interviewed  included:  Mary  Lee  Fay  and  Sean  McMullen  of  the  Oregon  Office 
of  Developmental  Disability  Services  (ODDS)  and  Rob  Horner,  Steve  Newton 
and  William  Ard  of  the  Specialized  Training  Program  at  the  University  of 
Oregon.  The  University  of  Oregon  was  involved  in  the  development  of  Oregon's 
outcome  monitoring  system  and  currently  is  under  contract  to  ODDS  to  perform 
data  analysis,  training  and  technical  assistance  along  with  the  Oregon  Technical 
Assistance  Center  (OTAC). 

Overview 

The  Residential  Outcome  System  (ROS)  was  developed  to  support  the  shift  from 
a  program-focused  system  of  assuring  quality,  to  more  person-centered 
measures.  In  the  mid  1980's,  the  Neighborhood  Living  Project  in  the  Specialized 
Training  Program  at  the  University  of  Oregon  developed  the  Valued  Outcome 
Information  System  (VOIS),  a  system  used  to  measure  accurate  information 
about  the  lifestyles  of  individuals  supported  by  ODDS.  VOIS  was  approved  by 
the  ODDS  for  use  in  residential  programs  licensed  to  provide  24-hour  residential 
support  services.  In  response  to  requests  for  a  more  streamlined  system  for 
programs  that  served  larger  numbers  of  individuals,  the  Oregon  Residential 
Lifestyle  Information  System  (ORLIS)  was  developed  in  the  late  1980's.  In  1993, 
the  VOIS/ORLIS  system  was  revised  into  one  system,  with  input  from  Oregon's 
residential  programs,  the  Neighborhood  Living  Project,  the  Oregon  Technical 
Assistance  Corporation,  and  the  Office  of  Developmental  Disability  Services. 
The  underlying  assumption  of  ROS  is  that  the  purpose  of  residential  services  is 
to  help  individuals  achieve  a  high  quality  lifestyle  in  which  there  is  a  match 
between  the  individual's  support  needs  and  the  support  the  individual  receives, 
as  well  as  a  match  between  the  individual's  lifestyle  preferences  and  the  lifestyle 
the  person  actually  needs. 

The  "valued  outcomes"  that  are  included  in  the  ROS  are: 

Physical  integration; 
Social  integration; 
Independence; 
Activity  preference;  and 
Activity  variety. 
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The  ROS  is  premised  upon  the  completion  of  an  Activities  Catalog,  a  listing  that 
includes  both  leisure  and  personal  management  activities.  The  overarching 
principle  upon  which  ROS  operates  is  that  there  must  be  consistency  between 
assessment,  program  development  and  evaluation  strategies.  ROS  provides  such 
consistency  by  having  the  Activities  Catalog  provide  the  basis  for  assessment, 
program  development  and  evaluation. 

According  to  the  researchers  at  the  University  of  Oregon,  one  of  the  most  critical 
factors  in  any  evaluation  system  is  that  it  provide  immediate  feedback  to  front- 
line staff.  Within  two  weeks  of  sending  in  their  data,  staff  are  provided  with  a 
summary  of  the  data.  This  reinforces  staff  in  a  way  that  encourages  them  to  use 
the  data  and  keep  collecting  it. 

Initially,  training  in  the  use  of  the  ROS  was  provided  by  the  Oregon  Technical 
Assistance  Center.  Currently,  the  providers  are  taking  responsibility  for  training 
their  own  staff  in  the  use  of  ROS.  Interestingly,  while  project  staff  were  in 
Oregon,  a  classified  ad  was  seen  for  direct  care  staff  that  specifically  stated, 
"experience  with  VOIS/ORLIS  preferred." 

Description  Of  Data  System 

Through  a  contract  with  the  Oregon  Office  of  Developmental  Disability  Services 
(ODDS),  ROS  data  are  maintained  by  the  Neighborhood  Living  Project  of  the 
Specialized  Training  Program  at  the  University  of  Oregon.  Residential 
providers  collect  ROS  data  on  a  quarterly  basis  and  it  is  forwarded  to  the 
University  of  Oregon  for  analysis  and  reporting.  The  Research  and  Evaluation 
Unit  of  the  Specialized  Training  Program  produce  a  report  every  six  months. 
The  data  collection  process  has  been  streamlined,  such  that  direct  contact  staff 
are  no  longer  responsible  for  graphing  the  data.  Therefore,  when  asked  how 
staff  feel  about  the  ROS,  must  staff  report  that  it  is  a  much  improved  system. 

Reliability 

A  study  of  Oregon's  outcome  assessments  was  conducted  by  J.  Stephen  Newton 
and  Robert  H.  Horner  of  the  University  of  Oregon.  Eighteen  programs  with  24- 
hour  staff  participated.  Programs  were  divided  into  four  cohorts  based  on 
geographical  clustering.  Cohort  A  consisted  of  four  programs;  Cohort  B 
consisted  of  6  programs;  Cohort  C  consisted  of  four  programs;  and  Cohort  D 
consisted  of  four  programs.  Three  of  the  four  programs  in  Cohort  D  were 
apartment  -based  as  opposed  to  beiog  group  homes.  All  the  programs  in  the 
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study  had  shift-type  staffing.  Thirty-three  participants  were  randomly  selected 
with  1  or  2  from  each  program.  Consent  was  obtained  from 
parents/guardians/advocates. 

The  four  major  dependent  variables  in  the  study  were: 

a)  physical  integration; 

b)  social  integration; 

c)  social  network  size;  and 

d)  social  network  structure. 

Physical  Integration  is  defined  as:  participation  in  activities  beyond  the 
boundaries  of  the  homes  of  individuals  with  disabilities;  the  average  length  of 
each  activity  was  at  least  15  minutes;  and  the  activities  involved  integration  of 
skills  which  had  utility  at  home  or  in  community  life.  Examples  of  such  activities 
are  grocery  shopping,  playing  basketball,  etc. 

The  Resident  Lifestyle  Inventory  (RLI)  was  one  of  the  techniques  employed  to 
measure  physical  integration.  The  RLI  was  used  to  interview  two  people  from 
the  staff  of  the  residential  program  who  were  well-versed  with  the  participants' 
daily  activities.  The  RLI  lists  144  activities.  The  interviewer  noted:  a)  the 
frequency  of  each  activity  in  the  last  30  days;  and  b)  whether  those  activities 
were  performed  at  home  or  in  the  community. 

Physical  integration  was  also  measured  by  direct  observation  of  the  participants' 
activities  with  the  utilization  of  the  "Valued  Outcomes  Information  System"  or 
VOIS.  A  staff  member  completed  an  "Activity  Tag"  for  each  participant.  The  tag 
recorded  the  name  of  those  with  whom  the  activity  was  performed;  the  name  of 
the  activity  itself;  and  an  indication  of  the  physically  integrated  nature  of  the 
activity.  The  VOIS  recorded  the  activities  weekly. 

The  second  variable  used  for  the  analysis  was  Social  Integration,  which  was 
operationalized  as  involvement  in  an  activity  with  individuals  outside  the 
program  excluding  the  residential  staff  and  other  participants  of  the  residential 
program.  The  first  measure  was  taken  from  the  Social  Network  Analysis  Form 
(SNAF)  which  required  the  participants  to  name  all  individuals  they  considered 
as  "socially  important"  within  their  a)  family;  b)  co-workers,  schoolmates,  other 
residents;  c)  people  paid  to  provide  service;  d)  friends;  and  e)  neighbors /others. 
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For  each  person  named,  the  respondent  was  asked  the  number  of  activities  they 
had  performed  with  them  in  the  last  thirty  days. 

The  second  measure  was  obtained  from  the  VOIS  Activity  Tags  which  produced 
the  number  of  activities  experienced  each  week  by  the  participants,  including 
those  in  the  home. 

Reliability  and  validity  of  the  data  were  assessed  using  the  following: 

VOIS:  Approximately  175  staff  members  produced  activity  tags  from  the  18 
participating  programs.  A  VOIS  workshop  was  conducted  to  train  the  staff  on 
coding  of  the  activity  tags.  The  staff  scored  85%  higher  following  the  training. 
An  inter  observer  tagging  accuracy  was  obtained  from  6  geographically 
convenient  programs.  One  coder  from  each  of  the  6  groups  with  a  score  of  95% 
and  above  was  designated  the  "master  coder".  The  master  coder,  with  other  staff 
members,  collected  20  tags:  a)  unobtrusively  with  at  least  80%  of  the  program 
staff;  b)  within  a  4-week  period;  and  c)  included  at  least  5  activities  which 
occurred  within  the  community.  In  total,  280  tags  were  completed  across  the  6 
reliability  programs. 

The  first  level  at  which  the  inter  observer  agreement  determined  the  reliability  of 
the  question  was  related  to  the  level  of  agreement  among  the  independent 
observers  with  respect  to  the  occurrence  of  physically  and  socially  integrated 
activities.  Within  a  range  of  85%-100%,  a  97.44%  agreement  was  reached  on 
whether  or  not  a  socially  or  physically  integrated  activity  had  occurred.  The 
agreement  level  was  99.24%  with  respect  to  agreement  on  the  physically 
integrated  activity  within  a  range  of  99.44%-100%.  Within  a  range  of  94.12-100%, 
the  average  for  agreement  on  socially  integrated  activity  was  98.84%. 

The  study  demonstrated  the  inter-rater  reliability  of  the  instruments  used. 
However,  there  are  certain  limitations  of  this  study.  Because  the  programs  were 
not  randomly  selected,  the  external  validity  of  the  study  is  questionable. 
Nevertheless,  this  limitation  is  partially  minimized  given  that  Oregon  has 
implemented  the  VOIS  feedback  system  in  approximately  200  programs  over  the 
past  years  indicating  positive  results. 

The  findings  are  limited  by  a  lack  of  a  continuous  direct  observation  by  unbiased 
research  staff  with  regard  to  interviews.  Nevertheless,  the  study  does  support 
previous  findings  indicating  positive  results  of  the  implementation  of  the 
feedback  system  irrespective  of  the  limitations  of  the  present  study. 
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Concurrent  Validity 


Concurrent  validity  has  been  established  between  the  VOIS  and  several  other 
instruments  that  measure  valued  outcomes  for  individuals.  The  instruments  that 
were  used  to  establish  concurrent  validity  with  the  VOIS  include  the  Behavior 
Development  Survey  (one  of  the  instruments  being  used  for  concurrent  validity 
measures  in  this  project),  and  the  Lifestyle  Map  (O'Brien).  The  concurrent 
validity  study  was  completed  by  J.  Stephen  Newton  at  the  University  of  Oregon 
(1986). 

In  terms  of  the  Lifestyle  Maps,  statistical  analyses  revealed  medium  to  large 
positive  correlations  between  ratings  on  the  Lifestyle  Maps  and  the  measures  on 
the  VOIS  instrument,  validating  the  measures  of  social  and  physical  integration. 
The  correlation  between  physical  integration  and  the  Community  Presence  Map 
was  .42  and  the  correlation  between  the  Community  Participation  Map  and 
social  integration  was  .47. 

With  regard  to  the  Behavior  Development  Survey,  statistically  significant 
correlations  were  obtained  between  the  VOIS  measure  of  social  integration  and 
the  BDS  factors  of  Personal  Self-Sufficiency  (.52),  Community  Self-Sufficiency 
(.47),  Personal  Social  Responsibility  (.49)  and  Personal  Adjustment  (.56). 

Applicability 

The  ROS  is  currently  used  in  community  programs  in  Oregon.  Although  some 
components  have  been  adopted  for  use  at  Fairview,  one  of  the  state's  large 
ICFs/MR,  the  ROS  in  its  entirety  is  not  being  implemented  in  any  large  setting. 
However,  this  observer  did  see  the  ROS  being  implemented  in  a  state-operated 
group  home  for  people  with  significant  medical  needs. 

Because  of  the  intensity  of  the  ROS  and  the  ongoing  nature  of  the  data  collection 
effort,  it  was  determined  that  bringing  it  to  Pennsylvania  to  examine  its 
applicability  across  settings  was  not  feasible.  In  order  to  perform  such  an 
exercise,  the  administration  and  staff  of  a  large  setting  would  have  to  be  invested 
in  the  process.  Strictly  for  scientific  inquiry,  it  would  be  virtually  impossible  to 
gain  the  cooperation  of  the  administration.  As  a  result,  it  is  questionable  as  to 
whether  this  instrument  could  be  used  in  a  large  facility  without  significant 
support  to  staff  to  maintain  the  level  of  effort  required  to  effectively  utilize  the 
system. 
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Discriminant  Validity 


The  Oregon  Office  of  Developmental  Disability  Services  was  contacted  to 
request  ROS  data  to  perform  the  discriminant  validity  portion  of  the  study. 
ODDS  approved  the  release  of  the  data,  and  a  file  was  electronically  transmitted 
to  Temple  from  the  Neighborhood  Living  Project  at  the  University  of  Oregon. 
The  data  file  include  970  individuals.  Unfortunately,  the  data  did  not  include 
an  acceptable  pool  of  "discriminating  variables"  such  as  type  of  living 
arrangement  or  level  of  disability.  The  data  file  that  was  transmitted,  due  to 
concerns  about  confidentiality,  did  not  have  identifying  information  which 
would  allow  us  to  "match"  the  outcome  data  with  our  data.  In  a  similar  vein, 
there  were  no  demographic  data  in  the  ROS  data  set.  Therefore,  it  was 
impossible  to  perform  a  discriminant  validity  analysis. 

Health  And  Safety 

During  the  site  visit  to  Oregon,  31  forms  were  collected  using  the  Temple 
University  Health  and  Safety  Assessment  developed  by  the  Temple  IOD. 
Because  the  ROS  is  only  used  consistently  in  community  settings,  all  data  were 
collected  in  community  settings  in  the  Eugene,  Salem  and  Portland  areas.  The 
intent  was  to  compare  data  from  the  Health  and  Safety  surveys  to  ROS  data  as 
well  as  to  ODDS  licensing  data.  Because  the  ROS  forwarded  to  Temple  by  the 
University  of  Oregon  data  not  have  identifying  information  even  at  the  provider 
agency  level,  it  was  impossible  to  use  the  data  for  Health  and  Safety 
comparisons.  Because  there  are  no  ICFs/MR  in  the  community,  comparisons 
were  not  made  with  ICF/MR  data. 

The  ODDS  licensing  data  were  forwarded  to  Temple  for  inclusion  in  the  Health 
and  Safety  analysis.  The  plans  of  correction  were  included  for  each  licensing 
review  and  were  used  as  a  source  of  data  for  this  analysis.  Numbers  of 
deficiencies  were  counted  in  the  health  and  safety  area  as  well  as  in  other  areas. 
Deficiencies  were  coded  into  three  areas:  1)  must  be  corrected  immediately;  2) 
must  be  corrected  within  30  days;  3)  must  be  corrected  in  90  days  or  at  a  time 
assigned  by  the  agency  self-assessment  team. 

In  order  to  analyze  these  data,  a  correlation  matrix  was  produced,  using  the 
Spearman  Correlation  procedure,  a  non-parametric  test.  Some  of  the  licensing 
items  were  correlated  with  the  data  collected  by  Temple.  The  number  of 
unusual  incidents  was  negatively  correlated  with  health  and  safety  deficiencies 
requiring  immediate  correction,  health  and  safety  deficiencies  requiring 
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correction  within  30  days,  and  other  deficiencies  requiring  immediate  correction. 
None  of  the  other  variables  was  significantly  correlated. 
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Conclusions 

The  new  version  of  the  ROS  presents  a  more  parsimonious  model  for  collecting 
data  than  was  provided- by  its  predecessors  (VOIS/ORLIS).  The  amount  of  data 
collection  and  analysis  required  of  direct  care  staff  is  lessened,  and  they  are,  we 
believe,  more  likely  to  collect  these  data  in  a  consistent  and  reliable  way. 
Additionally,  from  talking  with  staff,  it  is  obvious  that  the  data  are  used  in 
program  development  on  an  ongoing  basis.  The  ROS  is  demonstrated  to  be  a 
reliable  instrument  and  is  concurrently  valid  with  other  instruments  in  the  field 
used  to  measure  quality  outcomes  for  individuals  with  developmental 
disabilities. 

There  are,  however,  several  downsides  to  the  ROS.  Many  interviewees  continue 
to  complain  about  the  overly  quantitative  nature  of  the  assessments.  In  addition, 
there  is  no  linkage  between  the  ROS  data  and  licensing  survey  data.  As 
succinctly  stated  by  one  respondent,  "ROS  was  created  to  promote  quality, 
licensing  exists  only  to  prevent  atrocities."  While  some  may  view  this  as  a 
problem  with  the  instrument,  as  the  data  in  other  states  has  demonstrated,  the 
issues  of  health  and  safety  may  be  quite  different  and  distinct  from  those  related 
to  quality  outcomes. 
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UTAH  REPORT 


The  Utah  site  visit  took  place  November  27-29,  1995.  Individuals  interviewed 
included:  George  Kelner,  Sue  Geary,  Sue  Marquardt,  Georgia  Baddley,  Renae 
Taylor,  Robyn  Mendenhall,  Nonie  Lancaster,  Sylvia  Wright,  Sharon  Yearsley 
from  the  Utah  Division  of  Services  for  People  with  Disabilities,  Alan  Hayward  of 
the  Office  of  Licensing  Human  Service  Programs  within  the  Department  of 
Human  Services,  Ann  E.  Lee,  Tracey  Freeman  and  Helene  Middleton  from  the 
Department  of  Health  and  David  Hennessey,  a  provider  who  was  instrumental 
in  the  development  of  a  consumer  survey. 

Overview 

In  1990,  the  Department  of  Human  Services  in  Utah  regionalized.  This  shift  in 
responsibilities  and  operation  of  services  and  programs  for  people  with 
developmental  disabilities  resulted  in  the  need  for  the  Division  of  Services  for 
People  with  Disabilities  (DSPD)  to  refocus  its  quality  assurance  activities.  After 
a  review  and  analysis  by  HSRI,  it  was  recommended  that  the  Division  get  out  of 
the  business  of  licensing  and  focus  its  energies  on  review  and  support,  technical 
assistance  and  training.  A  Quality  Assurance  Academy  was  convened  consisting 
of  individuals  representing  consumers,  parents,  provider  and  advocacy 
organizations,  state  legislators,  case  managers,  Office  of  Social  Services  contract 
managers  and  regional  and  DSPD  managers.  The  Academy  recommended  that 
the  service  system  to  people  with  disabilities  be  driven  by  the  principles  of 
informed  choice,  community  membership /citizenship  and  individualization. 

In  response  to  the  recommendations  of  the  Academy  DSPD  developed  an 
Outcomes  Survey.  The  Outcomes  Survey  is  comprised  of  the  following  four 
standards: 

1.  The  provider  makes  available  for  the  individual  served  opportunities 
for  personal  growth  and  development  in  an  environment  which  uses  the 
least  restrictive  alternatives  available  which  are  compatible  with  the 
needs,  choices  and  goals  of  the  individual. 

2.  The  provider  facilitates  program  individualization  through  full 
participation  of  the  individual  in  planning  and  implementation  of  the  IPP 
and  training  to  allow  for  informed  decision  and  choice  making. 

3.  The  provider  enables  the  individual  to  fully  participate  in  Utah  life  and 
practice  a  normal  lifestyle  through  providing  opportunities  for  the 
individual  to  live,  work  and  associate  with  other  people,  including  those 
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with  and  without  disabilities,  using  services  readily  available  (generic)  in 
the  community. 

4.  The  provider  has  a  quality  assurance  system  in  place  which  assures 
that  services  are  provided  to  individuals  as  required  by  the  policies  and 
procedures  of  the  Department  of  Human  Services,  Division  of  Services  for 
People  with  Disabilities,  and  the  Title  XIX  Home  and  Community  Based 
Waiver;  which  meet  the  Department  of  Human  Services  Licensing  or 
Certification  Standards,  Division  of  Services  for  People  with  Disabilities 
standards,  and  all  the  Office  of  Social  Services  Contracting  requirements 
as  detailed  in  the  Attachment  E  of  the  Office  of  Social  Services'  contracts. 

The  first  three  standards  are  comprised  of  10  items  each.  Each  item  is  rated  on  a 
scale  of  one  through  six,  with  the  following  elements  considered  in  the  rating  of 
each  of  the  items:  method  of  assessing  interest  and  skills;  evidence  of 
opportunities  for  information;  evidence  of  education  and  training;  interagency 
coordination;  participation;  and  quality.  The  fourth  standard  is  simply  rated 
pass  or  fail.  The  total  possible  score  for  each  standard  is  60.  In  order  to  receive  a 
passing  score,  the  total  score  for  each  of  the  first  three  standards  must  be  at  least 
63%  of  the  possible  points,  or  a  score  of  38. 

If  an  agency  does  not  meet  criteria  on  one  of  the  standards,  a  plan  of  correction  is 
required;  a  follow-up  survey  is  then  performed  to  determine  whether  the  plan  of 
correction  has  been  implemented. 

The  Outcome  Surveys  are  completed  by  staff  of  the  Quality  Assurance  Unit  of 
DSPD.  The  instrument  was  developed  by  the  unit  and  training  in 
administration  of  the  survey  is  conducted  on-site  in  most  cases.  Two-hour 
training  sessions  are  also  held  periodically.  Case  managers  participate  in  the 
Outcome  Surveys  as  part  of  their  own  training. 

The  Outcome  Surveys  are  performed  for  individuals  living  in  community 
programs  that  are  funded  through  the  Home  and  Community  Based  Waiver  and 
those  individuals  in  100%  state-funded  placements.  However,  for  the  most 
capable  individuals,  (as  identified  by  scores  on  the  Instrument  for  Client  and 
Agency  Planning  (ICAP)  of  Level  1  or  2)  Outcome  Surveys  are  not  completed.  A 
consumer  satisfaction  survey  is  completed  and  if  the  individual  reports 
dissatisfaction,  an  Outcome  Survey  is  then  completed.  In  each  year  a  3% 
random  sample  is  drawn,  including  at  least  one  individual  from  each  provider 
agency. 

In  addition  to  the  Outcomes  Survey,  DSPD  has  had  a  consumer  survey 
developed  which  is  administered  to  10%  of  the  individuals  receiving  supported 
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living  services.  These  data  are  reported  in  narrative  form  and  are  not  amenable 
to  quantitative  analysis.  DSPD  also  sponsors  a  volunteer  monitoring  component 
in  which  DSPD  staff  train  the  monitors,  meet  regularly  with  the  leaders  of  the 
monitoring  group  and  address  concerns  raised  during  volunteer  monitoring 
visits. 

Description  Of  The  Data  System 

The  Outcomes  Survey  data  are  maintained  by  the  Quality  Assurance  Unit  of 
DSPD.  Reports  are  generated  on  an  annual  basis  analyzing  the  standards  by 
type  of  program,  ICAP  functioning  level  and  by  region.  Raw  scores  are  entered 
into  a  data  base  program  on  a  microcomputer. 

Reliability 

Reliability  data  are  collected  each  year  and  are  examined  and  utilized  to  target 
further  training  for  the  surveyors.  In  the  reliability  studies,  two  raters  complete 
scores  for  an  agency,  with  one  rater  does  the  actual  interviewing,  the  other 
observing.  In  the  most  recent  report  in  which  data  were  reported,  inter-rater 
reliability  scores  ranged  from  19%  to  78%.  This  presents  cause  for  some  concern. 
DSPD  identified  one  surveyor  as  having  particularly  low  scores  and  that 
individual  has  been  given  additional  training  and  opportunities  for  practice 
reviews.  The  results  of  the  reliability  study  are  presented  each  year  as  part  of 
the  annual  report.  Because  of  the  length  of  the  outcome  survey  process,  and  the 
requirement  for  intensive  training  in  the  application  of  standards,  it  was 
determined  that  a  reliability  study  by  the  project  was  not  feasible. 

Concurrent  Validity 

The  Division  took  the  Outcomes  Survey  to  the  State  of  Colorado  and 
administered  the  survey  in  six  sites.  In  five  of  the  six  sites  the  same  individuals 
were  residing  in  the  sites  as  when  the  Colorado  surveyors  performed  their  own 
surveys.  The  Utah  surveyors  rated  the  programs  on  a  Likert  scale  of  best  to 
worst  and  had  a  100%  agreement  with  the  Colorado  surveyors'  results  on  their 
instruments.  Unfortunately  these  data  have  not  been  reported  formally.  Again, 
because  of  the  length  of  the  outcome  survey  process  and  the  requirement  for 
intensive  training,  it  was  determined  that  further  study  of  the  concurrent 
validity  of  the  Utah  instrument  was  not  feasible. 
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Applicability 


The  Utah  instrument  has  been  used  only  in  HCBW  settings  and  in  settings  that 
are  100%  state  funded.  The  survey  is  not  used  in  private  or  public  ICF/MR 
facilities.  Neither  is  the  survey  used  in  settings  supporting  individuals  with  the 
highest  ICAP  scores,  i.e.,  individuals  who  are  the  most  capable,  as  it  is  seen  as 
unnecessarily  obtrusive.  Given  the  length  of  the  survey  and  the  need  for 
extensive  training,  it  is  not  feasible  to  administer  the  survey  in  Pennsylvania. 
The  outcome  survey  is  utilized,  however,  in  settings  supporting  people  with 
significant  disabilities,  people  who  use  wheelchairs,  and  people  with  significant 
communication  disabilities.  It  has  been  reported  by  Q.A.  Unit  staff,  however, 
that  the  instrument  could  possibly  be  used  in  ICFs/MR  with  some  additional 
training  of  staff.  The  major  reason  that  the  process  has  not  been  used  in  the 
ICF/MR  program  is  that  DSPD  does  not  have  jurisdiction  over  the  ICFs/MR; 
rather  they  are  under  the  auspices  of  the  Department  of  Health. 

Discriminant  Validity 

This  portion  of  the  review  determines  the  extent  to  which  the  Outcomes  Survey 
can  successfully  discriminate  between  individuals  based  on  their  scores.  An 
example  would  be  whether,  based  on  Outcomes  Survey  scores,  residential 
settings  can  be  predicted.  In  order  to  analyze  the  discriminant  validity  of  the 
instrument  the  most  recent  Outcomes  Survey  data  have  been  forwarded  to 
Temple,  and  the  analysis  have  been  performed.  Although  scores  are  attributed 
to  agencies,  the  raw  data  for  individuals  is  entered  onto  the  microcomputer.  The 
data  file  forwarded  to  Temple  for  analysis  included  data  for  84  individuals. 

In  order  to  assess  the  discriminant  validity  of  the  instrument,  a  discriminant 
function  analysis  was  performed.  The  purpose  of  discriminant  function  analysis 
is  to  statistically  distinguish  between  two  or  more  groups.  To  distinguish 
between  groups  a  collection  of  discriminating  variables  are  selected,  that 
measure  characteristics  on  which  the  groups  are  expected  to  differ.  Based  on 
discussions  among  the  Study  Team,  it  was  decided  that  the  two  variables  to  be 
used  as  discriminating  variables  would  be  level  of  disability  (in  this  case  the 
Instrument  for  Client  and  Agency  Planning  (ICAP))  and  living  arrangement. 
The  extent  to  which  differences  on  the  outcome  instrument's  scales  can  be 
explained  by  the  discriminating  variables  was  assessed.  The  outcome  indices 
that  were  examined  for  purposes  of  this  part  of  the  study  included: 

•  Opportunities  for  personal  growth  and  development; 

•  Program  individuaHzaHa*ihrough  participation  in  the  ISP; 
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•  Opportunities  for  community  inclusion;  and 

The  final  outcome,  existence  of  a  quality  assurance  system  was  not  included,  as 
it  is  not  a  scale;  rather  it  is  a  dichotomous  variable.  For  each  of  these  indices,  a 
discriminant  function  analysis  was  performed  using  the  discriminating  variables 
of  ICAP  level  and  living  arrangement,  to  see  whether  group  membership  (for 
both  ICAP  level  and  type  of  living  arrangement)  discriminated  scores  on  the  five 
outcome  scales.  For  purposes  of  this  study,  the  living  arrangement  variable 
includes:  day  training,  group  home,  professional  parent,  supervised 
apartment  and  employed.  For  purposes  of  this  analysis  the  breakdown  of  the 
sample  by  type  of  setting  is  as  follows: 


Type  of  Setting  Frequency  Percent 

Day  Training  30  36% 

Group  Home  14  17% 

Professional  Parent  10  12% 

Supervised  Apartment  25  30% 

Employed  5  6% 


Total 


84 


ICAP  level  for  the  group  is  as  follows: 


ICAP  Level 


Frequency 


Percent 


1 

2 
3 
4 
5 

MISSING 
TOTAL 


6 
16 
13 

5 

37 
5 

84 


8% 
21% 
17% 
6% 
48% 


The  first  set  of  analyses  examined  the  extent  to  which  type  of  living  arrangement 
was  successful  in  discriminating  between  scores  on  the  outcome  measures.  One 
of  the  ways  to  interpret  a  discriminant  function  analysis  is  that  once  a  set  of 
variables  is  found  which  provides  satisfactory  discrimination  for  cases  with 
known  group  memberships,  a  set  of  classification  functions  can  be  derived 
which  will  permit  the  classification  of  new  cases  with  unknown  memberships. 
For  example,  if  the  scores  on  a  given  outcome  variable  are  known,  if  the 
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discriminating  variables  work  well,  one  should  be  able  to  determine  from  which 
type  of  living  arrangement  the  individual  came  as  well  as  their  ICAP  level  based 
on  their  outcome  data.  As  a  check  of  the  adequacy  of  the  discriminant  functions, 
the  original  set  of  cases  can  be  classified  to  see  how  many  are  correctly  classified 
by  the  variables  being  used.  The  table  below  lists,  by  outcome  measure,  the 
percent  of  cases  correctly  classified  using  type  of  placement  as  a  discriminating 
variable. 

Percent  of  Cases  Correctly  Classified 
Using  Type  of  Living  Arrangement 

Opportunities  for  personal  growth  and  development  42.86% 
Program  individualization  through  participation  in  the  ISP  42.86% 
Opportunities  for  community  inclusion  26.19% 

In  a  variable  such  as  living  arrangement  where  there  are  five  possible  categories, 
one  would  expect  to  correctly  predict  group  membership  20%  of  the  time  by 
chance  alone.  As  the  table  above  demonstrates,  on  each  of  the  outcome 
variables,  we  were  able  to  predict  outcome  scores  by  living  arrangement  more 
often  than  would  be  expected  by  chance  alone. 

For  the  first  two  standards,  prediction  of  group  membership  is  twice  as  good  as 
would  be  expected  by  chance  alone.  For  the  third  standard,  opportunities  for 
community  inclusion,  prediction  of  group  membership  was  slightly  better  than 
by  chance  alone. 

The  second  set  of  analyses  examined  the  discriminating  properties  of  the 
variable,  ICAP  level.  The  table  below  lists,  by  outcome  measure,  the  percent  of 
cases  correctly  classified  using  ICAP  level  as  a  discriminating  variable. 

Percent  of  Cases  Correctly  Classified 
Using  Level  of  Mental  Retardation 

Opportunities  for  personal  growth  and  development  29.87% 
Program  individualization  through  participation  in  the  ISP  37.66% 
Opportunities  for  community  inclusion  26.19% 

Again,  in  a  variable  with  five  categories,  one  would  expect  correct  classification 
by  chance  alone  in  20%  of  the  cases.  As  the  table  demonstrates,  the  outcome 
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scores  correctly  predict  group  membership  by  ICAP  level  more  often  than  would 
be  expected  by  chance  alone,  for  all  three  outcome  indices. 

In  summary,  the  discriminant  validity  of  the  Utah  outcome  instrument  is  better 
than  would  be  expected  by  chance  alone,  for  both  type  of  living  arrangement 
and  ICAP  level. 

Health  And  Safety 

Temple  data  collectors  have  completed  health  and  safety  assessments  for  30 
individuals  in  Utah;  several  of  whom  live  in  private  ICFs/MR;  the  remaining 
individuals  living  in  a  variety  of  community  programs.  For  the  30  individuals 
for  whom  data  were  collected,  additional  data  were  obtained  from  the  DSPD  as 
well  as  other  offices  involved  in  surveying  health  and  safety  issues,  including 
the  Department  of  Licensing  within  the  Department  of  Human  Services,  and  the 
Division  of  Medicaid  program  Certification  in  the  Department  of  Health.  For 
those  individuals  living  in  ICFs/MR,  the  only  other  health  and  safety  data 
available  is  that  produced  through  the  ICF/MR  survey  in  the  form  of  an  Onsite 
Survey  and  Certification  Automated  Reports  (OSCAR).  The  OSCAR  data 
includes  the  actual  number  of  deficiencies  in  three  areas:  Conditions  of 
Participation,  Standards  and  Life  Safety  Code.  In  addition  to  providing  data  on 
actual  number  of  deficiencies  in  each  of  the  three  areas,  information  is  provided 
on  how  these  data  compare  to  the  national  average  number  of  deficiencies.  A 
scale  was  developed  for  each  of  the  three  areas,  comparing  the  facility  to  the 
national  average.  The  scale  ranges  from  1  to  5,  where: 

1  =  Much  worse  than  national  average  (more  than  2  points); 

2  =  Somewhat  worse  than  the  national  average  (2  points  or  less); 

3  =  The  same  as  the  national  average; 

4  =  Better  than  the  national  average  (2  points  or  less);  and 

5  =  Much  better  than  the  national  average  (more  than  2  points). 

Because  the  population  of  individuals  living  in  ICFs/MR  is  not  under  the 
jurisdiction  of  DPSD,  Outcome  Surveys  are  not  completed.  The  ICF/MR  survey 
is  the  only  quality  assurance  mechanism. 

For  those  individuals  living  in  smaller  settings,  there  are  several  other  health 
and  safety  mechanisms  in  place.    Wm  those  settings  where  four  or  more 
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individuals  live,  a  licensing  survey  is  completed  by  the  Division  of  Licensing 
within  the  Department  of  Human  Services.  Data  were  provided  on  the 
Residential  Treatment  Checklist  and  the  Life  Safety  Checklist.  In  the  case  of  each 
of  the  instruments,  the  total  number  of  deficiencies  was  collected. 

For  individuals  living  in  settings  of  fewer  than  four  people,  a  Certification 
Checklist  is  utilized;  on  that  instrument  total  number  of  deficiencies  were 
recorded  for  each  site.  Once  all  of  the  data  were  collected,  they  were  entered 
onto  a  spreadsheet  and  uploaded  onto  the  mainframe  computer  at  Temple  for 
analysis.  To  simplify  the  analysis,  the  residential  checklist,  certification  checklist 
and  life  safety  checklist  were  combined  to  form  one  variable  called  licensing.  On 
that  variable,  actual  number  of  deficiencies  were  reported.  Because  the  size  of 
the  sample  was  small  (there  were  only  12  individuals  for  whom  there  was 
individual  outcome  data),  non  parametric  statistics  were  used;  in  this  case 
Spearman  Correlations  were  computed  for  all  of  the  Health  and  Safety  data, 
along  with  the  actual  outcome  scores  on  the  first  three  indices  mentioned  above. 
Again,  the  existence  of  a  quality  assurance  system  (outcome  4)  was  not  included, 
because  it  is  a  dichotomous  variable  and  showed  very  little  variance. 

The  analysis  resulted  in  very  few  statistically  significant  correlations.  The  only 
outcome  measure  that  was  correlated,  was  the  third  outcome,  opportunities  for 
community  inclusion,  which  was  inversely  related  to  the  safety  scale  which 
included  inadequate  number  of  smoke  detectors,  no  fire  drill  in  the  past  60  days, 
etc.  (Rho=-.54,  p<.07).  The  state  health  and  safety  item  that  was  significantly 
correlated  was  the  licensing  item  which  included  the  number  of  deficiencies  on 
any  of  the  three  licensing  checklists.  The  licensing  deficiency  scale  was  inversely 
correlated  with  the  number  of  doctor's  visits  (Rho=-.59,  p<.008)  and  inversely 
correlated  with  the  safety  item  (Rho=-.43,  p<.06)  which  asks  about  neighborhood 
extremely  unsafe  neighborhood.  This  may  be  explained  with  the  assumption 
that  the  less  safe  the  neighborhood,  the  more  likely  the  site  was  to  have 
deficiencies  on  licensing  surveys. 
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Conclusions 


The  Quality  Outcomes  Survey  has  been  used  in  Utah  for  more  than  five  years. 
The  data  collection  has  facilitated  the  delivery  of  technical  assistance  by  DSPD 
and  the  data  have  been  used  as  part  of  a  "report  card"  that  is  produced  by  the 
Division  each  year.  The  survey  reflects  the  vision  and  mission  of  DSPD. 
However,  the  Outcomes  Survey  is  seen  as  cumbersome  by  many.  Reviews  are 
lengthy  and  given  the  sampling  strategy,  it  is  not  clear  that  they  are 
representative  of  an  agency's  performance.  In  addition,  the  Survey  is  only  used 
in  HCBW  and  state-funded  settings,  leaving  out  those  individuals  seen  as  most 
vulnerable  as  well  as  those  individuals  seen  as  most  capable. 
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The  outcome  assessment  instrument  may  be  reliable,  but  should  be  subjected  to 
a  more  formal  reliability  study  as  well  as  a  concurrent  validity  study  to  give  it 
more  credibility  in  the  field.  Regarding  health  and  safety,  the  outcomes 
assessment  process  does  not  seem  to  be  related  to  health  and  safety  in  any  major 
way.  The  separation  of  licensing  and  quality  assurance  functions,  however, 
seems  to  be  working  quite  well. 
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WYOMING  REPORT 


Introduction 

The  Wyoming  site  visit  took  place  on  November  30  through  December  2,  1995. 
An  initial  meeting  was  held  in  Cheyenne  between  project  staff  and  key  State 
agency  staff  for  the  purpose  of  informing  state  personnel  of  the  intent  of  the 
HCFA  project  as  well  as  providing  project  staff  with  an  overview  of  the  quality 
assurance  system  for  individuals  with  developmental  disabilities.  Temple 
University's  Institute  on  Disabilities  (IOD)  staff  met  with  the  University  of 
Wyoming  /UAP  staff  to  ascertain  the  availability  of  outcome  measurement 
instruments.  During  the  site  visit  IOD  staff  also  collected  health  and  safety 
information  on  32  individuals  receiving  residential  services  from  the  Division  of 
Developmental  Disabilities  (DDD)  in  Cheyenne,  Riverton,  and  Lander.  The 
following  individuals  were  interviewed  during  the  site  visit: 

•Robert      Clabby,      Division      of      Developmental  Disabilities 
Administrator/Wyoming  State  Training  School  Superintendent 

•Jon  Fortune,  Ed.D.,  DDD,  Adult  Services  Manager 

•Ken  Heinlein,  Ed.D.,  University  of  Wyoming/UAP 

•Nancy  Riley,  DDD  Resource  Specialist,  Cheyenne 

•Sharon  Kelsy,  Magic  City  Enterprises,  Cheyenne 

•Ramona  Doidge,  DDD  Resource  Specialist,  Riverton /Lander 

•David  Brunner,  WSTS,  Quality  Assurance  Coordinator 

•Michael  Aurand,  WSTS  QMRP 

•Shawn  Griffin,  Community  Entry  Services  (CES),  Chief  Operations  Officer 
•Joe  Hoffman,  CES  Community  Living  Director 

Contact  with  key  stakeholders  in  Wyoming  by  IOD  staff  was  continuous  as 
systems  for  analysis  of  the  health  and  safety  data  were  developed  and  the  need 
for  securing  additional  information  was  identified. 
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Overview 


Wyoming  is  a  vast,  sparsely  populated,  rural  state  which  has  nine  major 
community  providers  and  one  large  state-run  ICF/MR,  Wyoming  State  Training 
School  (WSTS).  One  hundred  and  forty  one  individuals  reside  in  WSTS,  with 
approximately  915  individuals  being  supported  in  the  community.  Of  those 
being  served  in  the  community,  740  adults  and  children  are  supported  through 
the  HCB  waiver.  State  officials  are  very  proud  of  the  fact  that  they  have  no 
community  based  ICF/MR  programs. 

In  1990,  the  Wyoming  Protection  and  Advocacy  System,  Inc.,  filed  a  class  action 
suit  (Weston,  et  al.  v.  Wyoming  State  Training  School,  et  al.)  against  the  state  on 
behalf  of  all  individuals  with  mental  retardation,  currently  at  the  WSTS,  and 
those  individuals  at  risk  of  being  placed  at  WSTS.  A  Consent  Decree, 
negotiated  in  1991  precipitated  an  accelerated  rate  of  community  placement  for 
residents  of  WSTS;  the  establishment  of  an  Independent  Case  Management 
system;  and  a  requirement  that  the  division  draft  quality  assurance  standards. 
The  division  has  published  a  report  outlining  all  quality  assurance  efforts  in 
community  services  for  persons  with  developmental  disabilities  and  mental 
retardation  since  1991.  A  Settlement  Agreement  replaced  the  Consent  Decree  in 
January  1995  maintaining  the  State's  requirement  for  quality  assurance  efforts 
and  Independent  Case  Management  Services. 

State  legislation  enacted  in  1982  was  initiated  by  community  providers  to  require 
the  Commission  on  Accreditation  of  Rehabilitation  Facilities  (CARF)  to  accredit 
community  programs.  As  of  the  site  visit  date,  all  nine  of  the  major  community 
providers  had  attained  three  year  accreditation  status,  (the  longest  period  of  time 
an  agency  can  receive  accreditation  from  CARF).  The  community  providers  are 
required  under  CARF  standards  to  complete  their  own  internal  program 
reviews.  No  Wyoming  agency  is  scheduled  to  undergo  the  new  outcome  based 
CARF  accreditation  process  until  1997. 

In  addition  to  CARF  accreditation,  the  Wyoming  Division  of  Developmental 
Disabilities  conducts  an  annual  program  review  for  each  of  the  provider 
agencies.  A  team  of  four  to  six  individuals  from  the  DDD  and  the  Governor's 
Planning  Council  on  Developmental  Disabilities  review  all  aspects  of  an 
agency's  programs,  and  issues  a  narrative  report.  Particular  attention  is  given  to 
any  of  the  CARF  standards  in  which  an  agency  was  deficient.  The  state  review 
monitors  the  agency's  action  plan  to  correct  the  deficiency  during  their  annual 
review. 
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Wyoming  has  been  using  the  Inventory  for  Client  and  Agency  Planning 
instrument  (ICAP)  (Bruininks,  Hill,  Weatherman,  &  Woodcock,  1986)  to 
measure  the  adaptive  behavior,  functional  characteristics  and  challenging 
behaviors  of  the  individuals  it  provides  supports  to,  since  1988.  In  1994, 
Coming  Home  on  the  Range:  Changing  ICAP  Profiles  of  the  People  Served 
and  Their  Community  and  Institutional  Services  in  Wyoming,  1988-1994 
(Fortune,  Heinlein,  Martinez  &  Fortune)  compared  the  statewide  data  set  from 
1988  to  1994.  The  study  found  an  increase  in  the  number  of  persons  being 
supported  in  the  community  instead  of  the  institution,  "...164  individuals  with  a 
variety  of  disabilities,  requiring  different  levels  of  support  are  now  being  served 
in  the  community  programs  instead  of  the  state  institution.  There  are  also  116 
people  served  in  the  community  during  that  time  who  were  diverted  from 
institutional  care."  The  research  further  demonstrated  "that  individuals  with  a 
wide  range  of  needs  can  be  served  in  community  programs  in  a  profoundly 
rural  state  if  provided  with  a  choice  of  well  funded  services  that  the  people  who 
are  served  need." 

The  Resident  Lifestyle  Inventory  (RLI),  is  a  survey  instrument  developed  in 
Oregon  to  monitor  the  extent  of  an  individual's  participation  in  community 
activities.  This  instrument  was  administered  to  people  receiving  residential 
supports  under  the  auspices  of  Wyoming's  DDD  for  the  month  of  July  1993. 
Independent  Case  Managers  conducted  interviews  with  class  members  and  a 
knowledgeable  informant.  Information  was  collected  about  the  array  of 
activities  class  members  (N=99)  were  engaged  in  during  the  30  days  prior  to  the 
interview.  (Collecting  the  data  retrospectively  as  Wyoming  did  differs  from  the 
RLI's  use  in  Oregon.  Information  is  collected  using  the  RLI  in  Oregon  as  it  is 
occurring,  on  a  weekly  basis).  The  Independent  Case  Managers  also  interviewed 
two  staff  members  from  each  Regional  Service  Provider  and  from  WSTS  (N=21). 
The  data  gathered  from  these  two  groups  was  compared  and  different  patterns 
of  activities  for  each  group  did  emerge.  On  average,  staff  engaged  in  more 
activities  than  class  members.  There  are  no  plans  to  use  this  instrument  again,  as 
it  was  perceived  to  be  too  time  consuming. 

A  Consumer  Satisfaction  Survey  was  conducted  for  three  years,  from  1992  to 
1994,  as  mandated  by  the  Consent  Decree.  The  surveys  were  sent  to  primary 
contacts  of  a  sample  of  adult  class  members  and  non  class  members.  Responses 
were  received  in  1994  from  109  primary  contacts.  The  questionnaire  consisted  of 
30  questions  asking  the  primary  contact  to  rate  the  quality  of  services.  The 
response  rate  for  the  two  groups  was  different,  "Class  members  had  a  54%  (61  of 
113)  response  rate  and  non-class  members  had  a  60%  (48  of  80)  response  rate, 
when  returns  for  incorrect  addresses  were  taken  into  account.  The  effective 
response  rate  was  56.5%,  which  is  about  ten  percent  less  than  the  1993  survey, 
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but  above  the  minimum  response  rate  of  50%  commonly  regarded  for  valid 
results/'  (1994  Consumer  Survey  Results,  Ken  B.  Heinlein,  Ph.  D.) 

Wyoming's  Protection  and  Advocacy  Agency  developed  a  10  question  survey 
instrument  to  be  used  for  collecting  information  on  Regional  Service  Providers. 
This  instrument  focuses  primarily  on  the  existence  of  agency  policies  and 
procedures  for  ensuring  the  rights  of  individuals  receiving  supports  from  the 
providers.  The  instrument  also  documents  an  agency's  in-service  training 
programs. 

ARK  Regional  Services,  a  Wyoming  regional  provider  has  developed  a 
copyrighted  Outcome  Management  System  called  SEE.  SEE  is  an  acronym  for 
Satisfaction,  Effectiveness,  and  Efficiency.  These  principles  are  core 
requirements  of  the  CARF  accredited  Outcome  Management  System.  The 
system  is  available  for  use,  free  of  charge,  by  Wyoming  agencies.  ARK  Regional 
Services  also  plans  to  market  their  system  nationally. 

The  University  of  Wyoming/UAP  has  designed  a  Quality  of  Life  Questionnaire 
to  gather  data  related  to  the  DDE^s  identified  outcomes.  The  UAP  has 
submitted  a  proposal  to  the  Division  to  collect  data  on  a  15%  sample  of  adults 
with  developmental  disabilities  receiving  HCB  waiver  services.  Included  in  this 
proposal  would  be  a  modified  version  of  the  consumer  satisfaction  survey  used 
in  1992  through  1994.  This  survey  would  be  mailed  to  the  primary  contact 
person  of  the  adults  receiving  HCB  waiver  supports.  The  UAP  is  hoping  for  FY 
'97  funding  of  this  project. 

Description  Of  The  Data  Base 

The  following  components  of  Wyoming's  quality  assurance  system  are  available 
on  various  data  bases  and  computer  programs.  ICAP  information  is  available  as 
an  ASCII  file.  One  month's  data  on  the  RLI  is  available  on  the  SPSS  system. 
Data  from  the  Primary  Contact  survey  is  also  available  on  SPSS.  The  State  survey 
is  available  on  a  word  processing  file.  The  CARF  data  is  not  maintained  on  a 
computer  program. 
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Reliability 


There  is  not  one  viable  outcome  measure  instrument  that  is  being  used  in 
Wyoming  at  this  time,  therefore,  the  Study  Team  has  determined  that  a 
reliability  study  id  virtually  impossible  to  conduct. 

Concurrent  Validity 

There  is  not  one  viable  outcome  measure  instrument  that  is  being  used  in 
Wyoming  at  this  time,  therefore,  the  Study  Team  has  determined  that  a  study  of 
concurrent  validity  is  virtually  impossible  to  conduct. 

Applicability 

There  is  not  one  viable  outcome  measure  instrument  that  is  being  used  in 
Wyoming  at  this  time,  therefore,  the  Study  Team  has  determined  that  we  are 
unable  to  establish  any  properties  of  applicability. 

Discriminant  Validity 

There  is  not  one  viable  outcome  measure  instrument  that  is  being  used  in 
Wyoming  at  this  time,  therefore,  the  Study  Team  has  determined  that  we  are 
unable  to  determine  any  discriminant  validity  properties. 

Health  And  Safety 

Health  and  Safety  data  were  collected  on  32  individuals  residing  in  seven 
community  residences  and  the  Wyoming  State  Training  School.  Two  regional 
provider  agencies  were  represented  in  this  sample.  This  information  was 
collected  in  the  state  capital,  Cheyenne,  which  is  located  in  the  southeastern 
portion  of  the  State  and  toward  the  middle  of  this  vast  state  in  the 
Riverton/ Lander  area.  The  data  were  collected  on  November  30  and 

December  1  and  2  by  IOD  staff  using  the  Temple  University  Health  and  Safety 
Issues  Instrument,  developed  by  the  Temple  University  Institute  on 
Disabilities/UAP. 
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The  health  and  safety  data  collected  at  WSTS  was  compared  with  OSCAR  data. 
The  community  residence  health  and  safety  data  collected  with  theTemple 
University  Health  and  Safety  instrument  was  compared  with  the  CARF 
accreditation  data  completed  for  each  agency  in  1994  using  the  1993  CARF 
standards.  A  further  analysis  of  the  health  and  safety  data  collected  at  the 
community  sites  was  originally  contemplated  by  the  Study  Team.  Our  intention 
was  to  compare  the  health  and  safety  data  with  the  findings  contained  in  the 
state  reports  as  part  of  their  annual  site  review.  The  state  site  reviews  are 
reported  in  narrative  form  that  is  not  quantifiable.  Therefore,  because  the 
licensing  information  is  too  qualitative  in  form,  we  are  unable  to  analyze  this 
information  and  compare  it  with  the  health  and  safety  data. 

The  CARF  Accreditation  Site  visits  for  the  two  agencies  in  Wyoming  where 
HCFA  health  and  safety  data  was  collected  were  conducted  by  CARF  using  the 
1993  CARF  standards.  These  standards  are  not  outcome  measures  but  are  used 
as  part  of  a  process  system  of  accreditation.  The  results  of  the  accreditation 
process  results  in  accreditation  of  either  1,  2,  or  3  years.  Both  agencies  were 
accredited  for  3  years.  The  CARF  Survey  Report  is  the  document  produced  by 
the  CARF  Survey  Team  which  is  reported  in  narrative  form  and  is  organized  by 
separate  Organizational  and  Programmatic  Sections;  e.g.  under  the  Specific 
Program  Standards  section  the  programs  an  agency  provides  are  delineated  and 
a  narrative  report  citing  particular  standards  are  listed. 

CARF  uses  four  possible  ratings  for  the  1993  standards.  The  ratings  are 
Conformance  with  Commendation,  Conformance,  Conformance  with 
Suggestion,  and  Non-Conformance  with  Recommendation.  The  following  rating 
scale  has  been  developed  for  the  CARF  ratings. 

Conformance  with  Commendation  =4 

Conformance^ 

Conformance  with  Suggestion=2 
Non-Conformance  with  Recommendation=l 

In  order  to  conduct  a  comparison  of  the  Temple  University  health  and  safety 
information  and  the  ratings  contained  in  the  CARF  Survey  Report  the  Study 
Team  identified  4  applicable  sections  in  each  of  the  agencys'  CARF  Survey 
Reports.  These  sections  were  identified  as  being  applicable  for  analysis  in  that 
they  contained  criteria  which  could  be  matched  under  the  eight  HCFA 
outcomes.  The     CARF  sections  are  Physical  Facilities,  Health  &  Safety; 
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Assessment;  Individual  Plan  Development,  Implementation,  and  Management; 
and  Community  Living  Programs:  Integrated. 

In  addition  to  identifying  the  total  number  of  applicable  standards  in  each  of  the 
four  sections  we  applied  the  rating  scale  mentioned  above  to  each  section  and 
established  the  range  of  scores  possible  that  an  agency  could  attain  for  each  of 
the  sections.  The  following  table  displays  the  identified  sections;  the  number  of 
applicable  standards  and  the  possible  range  of  scores  : 

Applicable  CARF  RptSections      #  of  Standards        Range  of  scores 


Physical  Facilities,  Health  &  Safety  8  8-32 

Assessment  3  3-12 
Individual  Plan  Development,  Implementation, 

and  Management  4  4-16 

Community  Living  Programs:  Integrated  12  12-48 


In  the  analysis  of  the  data  we  conducted  non  parametric  correlations  and 
assessed  the  Spearman  Correlation  Coefficients  and  found  no  significant 
correlations.  This  in  part  is  due  to  the  small  number  of  cases  in  the  study.  In 
addition,  it  is  possible  that  due  to  the  passage  in  time  since  the  CARF  data  were 
collected,  and  the  difference  in  the  dimensions  assessed  by  the  Health  and  Safety 
Instrument  and  CARF,  there  are  no  relationships  between  the  variables 
measured. 

A  d  d  r  •  •  •  Ol  0  2      0  3      t       KID      M  B  0  H  D      IDOC      U   I  BP  SAPISCSAPIHfcS  A  •  •  •  I  I   P  P  L  i  V  Pt 

Agency!  4         1         4         0.(4  0      2  «         0       2         01  1       9  0 

A  g  •  n  c  y      2  0  0         C  .  C  0  0  0.5  0  23  10  14  34 

A  g  •  n  c  y      3  0.0  1  0.2         9      4  0  0         0      6  3  0  27  14  11  35 


Conclusions 


Wyoming  has  made  very  strong  initial  steps  toward  measuring  quality  of 
supports  and  services  to  their  citizens  with  developmental  disabilities.  The 
participation  of  community  providers  in  the  CARF  accreditation  process  over 
the  past  decade  indicates  a  strong  desire  to  strive  for  the  delivery  of  quality 
supports.  Their  search  to  measuring  quality  is  ongoing,  and  they  have  tried  a 
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number  of  approaches  such  as  the  use  of  the  RLI  for  a  month  which  was  never 
continued  primarily  because  of  the  perceived  excessive  amount  of  person  hours 
necessary  in  collecting  the  data.  The  UAP 's  proposal  to  measure  outcomes 
which  would  include  a  modification  of  the  1992-1994's  primary  contact 
satisfaction  survey  is  an  indication  of  the  state's  evolution  toward  measuring 
quality  on  an  outcome  basis.  However  at  present  there  is  no  systemic  effort. 
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REPORT  ON  AC  SURVEY  REPORT 


Overview 

The  purpose  of  this  visit  was  to  observe  the  implementation  of  the  Accreditation 
Council's  new  Outcome-based  Performance  Measures  (1993).  The  design, 
development  and  adoption  of  the  Outcome-based  Performance  Measures 
involved  a  two-year  process  which  began  in  1991.  Through  focus  groups  and 
individual  interviews  with  individuals  with  disabilities,  outcomes  were 
identified  that  people  expect  from  the  services  and  supports  they  receive.  The 
Council  received  input  on  successive  drafts  of  the  proposed  measures  from 
people  with  disabilities,  families,  service  providers,  representatives  of  state  and 
federal  agencies  and  other  professionals.  The  outcome  measures  were  adopted 
for  use  in  accreditation  reviews  in  July,  1993.  The  Council  conducted  field  tests 
of  the  standards  prior  to  their  final  adoption.  The  purpose  of  the  field  tests  was 
to  insure  that  the  measures  were  valid,  reliable  and  applicable  across  settings. 
The  field  tests  occurred  in  ICFs/MRs,  supported  living,  supported  employment 
and  service  coordination  settings. 

Prior  to  the  adoption  of  the  Outcome  standards,  the  accreditation  process 
focused  on  organizational  activities,  resources  and  services.  The  measures  of 
assessment,  planning  and  training  addressed  quality  in  terms  of  what  the 
organization  had  achieved,  not  in  terms  of  outcomes  for  the  person  receiving 
services.  This  shift  represents  a  significant  change  in  the  way  accreditation  now 
occurs.  The  individual  receiving  services  and  supports  now  dictates  what  is 
measured  and  what  is  observed. 

The  Survey  Outcomes  and  Process 

As  a  result  of  the  work  done  by  the  Accreditation  Council,  the  following 
outcomes  were  identified  for  individuals: 

1.  People  choose  personal  goals. 

2.  People  realize  personal  goals. 

3.  People  choose  where  and  with  whom  they  live. 

4.  People  choose  where  they  work. 

5.  People  choose  how  to  use  their  free  time. 

6.  People  choose  services. 

7.  People  choose  their  daily  routine. 
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8.  People  participate  in  the  life  of  the  community. 

9.  People  interact  with  other  members  of  the  community. 

10.  People  perform  different  social  roles. 

11.  People  have  friends. 

12.  People  remain  connected  to  natural  support  networks. 

13.  People  have  intimate  relationships. 

14.  People  exercise  rights. 

15.  People  are  afforded  due  process  if  rights  are  limited. 

16.  People  are  free  from  abuse  and  neglect. 

17.  People  are  respected. 

18.  People  have  time,  space  and  opportunity  for  privacy. 

19.  People  have  and  keep  personal  possessions. 

20.  People  decide  when  to  share  personal  information. 

21.  People  have  health  care  services. 

22.  People  have  the  best  possible  health. 

23.  People  are  safe. 

24.  People  use  their  environments. 

25.  People  live  in  integrated  environments. 

26.  People  have  economic  resources. 

27.  People  have  insurance  to  protect  their  resources. 

28.  People  experience  continuity  and  security. 

29.  People  are  satisfied  with  services. 

30.  People  are  satisfied  with  their  personal  life  situations. 

In  addition  to  the  individual-specific  outcomes,  The  Accreditation  Council  also 
identified  Performance  Measures  for  Organizations.  These  include: 

Personal  Health,  Safety  and  Welfare 

1.  The  organization  protects  the  rights  of  people. 

2.  The  organization  demonstrates  a  commitment  to  using  positive 
approaches  in  all  services  and  support  activities. 

3.  The  organization's  service  practices  and  staff  demonstrate  sensitivity 
and  concern  for  personal  dignity  and  respect. 

4.  The  organization  implements  procedures  for  investigation  and 
intervention  in  all  instances  of  alleged  abuse  and  neglect. 

5.  The  organization  owns,  operates  or  leases  buildings  that  comply  with 
all  applicable  fire  and  sanitation  codes. 

6.  The  organization  implements  procedures  for  meeting  all  emergencies, 
such  as  fire,  severe  weather  and  health. 

7.  The  organization  implements  employment  screening  procedures  that 
minimize  unnecessary  or  unreasonable  risk. 
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Fiscal  Management 


1.  The  organization  has  a  budgeting  and  accounting  system. 

2.  The  organization  has  an  independent  audit  of  its  fiscal  activities 
annually. 

3.  The  organization  has  separate  accounting  for  funds  managed  for 
people. 

Human  Resources  Management 

1.  The  organization's  personnel  practices  meet  all  state  and  federal  fair 
labor  regulations. 

2.  The  organization  provides  opportunities  for  staff  training  and 
personnel  development. 

3.  The  organization  regularly  evaluates  and  provides  feedback  to  its  staff 
on  their  performance. 

Planning  and  Evaluation 

1.  The  organization  has  a  clear  statement  of  its  mission  and  purpose. 

2.  The  organization  conducts  an  ongoing  evaluation  of  success  in 
achieving  desired  outcomes. 

3.  The  organization  includes  input  and  involvement  from  people  served 
and  others  in  its  evaluation  and  planning  activities. 

4.  The  organization  implements  a  program  for  continuous  quality 
improvement 

An  agency  files  an  application  for  accreditation  review  approximately  six  to 
twelve  months  prior  to  the  review.  The  application  includes  information  about 
the  services  and  supports  the  agency  provides  and  information  about  the  people 
to  whom  the  agency  provides  supports  (gender,  age,  type  of  disability).  The 
third  section  of  the  application  includes  information  about  several  of  the 
Performance  Measures  for  Organizations.  This  enables  the  Accreditation 
Council  to  determine  whether  the  agency  is  ready  for  an  on-site  review,  or 
whether  the  agency  would  be  better  served  by  first  receiving  consultation  in 
areas  of  perceived  need  for  improvement. 

In  addition  to  the  aforementioned  information,  the  agency  supplies  The 
Accreditation  Council  with  a  roster  of  the  individuals  supported.  This  assists  in 
the  selection  of  a  "representative"  group  of  individuals  to  be  interviewed  during 
the  on-site  review.  In  the  case  of  this  agency  which  supports  23  individuals  in 
apartments,  15  individuals  in  an  ICF/MR,  and  30  individuals  in  two  15  person 
group  homes  that  are  certified  under  the  Home  and  Community-based  Waiver, 
one  individual  was  selected  from  each  of  the  group  homes,  one  from  the 
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ICF/MR  and  four  individuals  were  selected  from  the  apartment  program,  for  a 
total  of  seven  individuals  comprising  the  sample. 

The  length  of  the  on-site  review  is  determined  by  the  number  of  people 
supported  by  the  organization,  the  scope  of  services  and  supports  provided  and 
the  size  of  the  geographic  area  in  which  supports  and  services  are  provided.  The 
Accreditation  Council  determines  the  number  of  surveyors  and  the  number  of 
days  needed  to  complete  the  on-site  review  process.  For  the  review  observed, 
there  were  two  surveyors  for  four  days  on  site. 

Accreditation  Council  surveyors  are  skilled  professionals  in  the  field  of 
developmental  disabilities.  Surveyors  must  have  significant  experience  in 
working  directly  with  individuals  with  developmental  disabilities.  It  is  not  a 
requirement,  however,  that  surveyors  work  in  accredited  facilities.  Of  the  two 
surveyors  on  this  review,  one  worked  in  state  government  and  the  other  worked 
in  a  large,  public  ICF/MR.  Both  of  the  surveyors  had  participated  in  many 
reviews  under  the  1990  Standards,  and  both  had  participated  in  multiple 
reviews  under  the  new  outcome-based  standards.  The  Council  designated  one 
of  the  surveyors  as  the  team  leader. 

Once  the  application  is  completed  and  deemed  acceptable  for  on-site  review,  the 
agency  receives  a  set  of  materials  to  guide  them  in  a  self-assessment  process.  In 
the  self-assessment  process,  the  agency  selects  at  least  three  individuals  to 
interview,  and  for  each  individual  determines  whether  each  of  the  outcomes  is 
present,  and  also  makes  a  determination  of  whether  there  is  a  process  in  place  to 
support  the  outcomes.  The  self-assessment  is  forwarded  to  The  Accreditation 
Council  approximately  6-8  weeks  prior  to  the  on-site  review.  In  this  particular 
situation,  the  agency  had  never  been  surveyed  using  the  outcome  standards.  As 
a  result,  they  requested  that  one  of  the  individuals  interviewed  for  the  self 
assessment  also  be  part  of  the  on-site  sample.  This  would  let  the  agency  know  in 
a  sense,  whether  they  had  been  "too  hard"  or  "too  easy"  on  themselves  for  the 
self-assessment.  It  is  a  practice  of  The  Accreditation  Council  to  include  in  the 
sample,  at  least  one  individual  who  was  a  part  of  the  self-assessment  process. 

The  Presence  and  Role  of  Observers  During  the  On-Site  Review 
Process  (The  Accreditation  Council's  Policy) 

The  Council  routinely  sponsors  its  Board  members  and  potential  new  staff 
members  to  observe  Accreditation  Reviews.  The  purpose  of  observation  by 
Board  Members  is  to  assist  them  in  their  responsibilities  as  Council  policymakers 
by  ensuring  their  direct  familiarity  with  the  Review  process  and  the 
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implementation  of  the  Outcome  Based  Performance  Measures.  The  purpose  of 
observation  by  potential  and  new  staff  members  is  to  cooperatively  assess  their 
skills  and  interest  in  becoming  reviewers /trainers  and  to  increase  skills  of  those 
already  conducting  reviews  or  providing  training  and  consultation. 

Observers  typically  participate  in  the  Review  by  accompanying  members  of  the 
Review  Team  as  they  complete  their  activities:  interviewing  people  participating 
in  services,  discussions  with  staff  members,  observations,  record  reviews,  and 
assessment  of  The  Performance  Measures  for  Organizations.  There  are  situations  in 
which  the  observer  is  assigned  responsibility  for  particular  aspects  of  the  review 
in  order  to  provide  experience  and  the  opportunity  to  determine  skill  and 
comfort  level  with  the  process.  This  occurs  at  the  discretion  of  the  Team  Leader 
in  conjunction  with  the  Director  of  the  Accreditation  Department  and  is  decided 
based  on  the  purpose  of  the  observation.  No  decisions  about  any  aspect  of  the 
Review  are  made  by  an  observer.  As  in  any  Review,  decisions  are  made  only 
after  thorough  discussion  of  information  by  the  Team. 

Observers  participate  in  Reviews  in  the  spirit  of  The  Council's  mission  to 
improve  quality  through  a  cooperative  and  consultative  relationship. 

Observation  of  the  Survey  Process 
Day  One 

On  the  first  day  of  the  review,  the  team  members  met  with  the  key  staff  from  the 
agency,  including  the  executive  director.  This  gave  everyone  at  the  agency  an 
opportunity  to  introduce  themselves.  The  surveyors  introduced  themselves,  as 
did  the  Temple  observer,  explaining  the  role  of  the  observer  in  the  process.  The 
Team  Leader  gave  an  overview  of  the  AC  on-site  review  process.  The  staff  were 
able  to  provide  additional  information  about  programs  and  services,  ask 
questions  and  assist  in  the  establishment  of  the  schedule.  At  that  time  the 
sample  selection  was  done,  to  purposely  include  one  of  the  individuals  who  was 
part  of  the  self-assessment.  The  team  was  also  informed  at  that  time  that  six  of 
the  individuals  supported  by  the  agency  did  not  wish  to  participate  in  the  on- 
site  reviews.  As  is  the  Accreditation  Council  policy,  those  individuals  were 
removed  from  consideration  for  sample  selection  purposes. 

The  purpose  of  the  sample  selection  is  to  attempt  to  reflect  the  various 
characteristics  of  the  individuals  supported  by  the  agency.  In  addition,  an 
attempt  is  made  to  select  individuate  receiving  various  types  of  residential, 
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employment  and  other  services  and  supports  provided  by  the  agency.  The 
sample  for  this  review  included  seven  individuals. 

Once  the  sample  selection  was  completed,  the  individual  interviews  began.  At 
that  point,  the  two  reviewers  split  up,  each  interviewing  different  individuals. 
The  Temple  observer  accompanied  one  of  the  surveyors.  The  first  individuals 
interviewed  preferred  to  be  interviewed  at  the  main  office  of  the  provider 
agency,  which  is  also  the  location  of  the  agency-supported  workshop.  The 
people  were  interviewed  along  with  the  individuals  who  supported  them,  at  the 
request  of  the  individuals,  and  included  direct  support  staff  from  the  day 
program  and  residence,  and  program  supervisors.  Each  of  the  interviews  lasted 
approximately  one  hour.  The  interviews  were  conducted  using  the  protocols 
and  questions  developed  by  The  Accreditation  Council.  Follow-up  questions 
were  asked  at  the  discretion  of  the  team  members. 

To  determine  inter-rater  reliability,  The  Accreditation  Council  has  established  a 
policy  that  team  members  both  interview  one  individual,  with  one  team  member 
conducting  the  interview  and  taking  notes,  with  the  other  team  member 
observing  and  taking  notes.  Both  team  members  participate  in  all  follow-up  and 
observations.  After  all  information  has  been  gathered,  both  team  members 
independently  assess  the  presence  or  absence  of  each  of  the  outcomes  and 
processes.  The  inter-rater  reliability  review  was  completed  on  the  first  day. 

On  the  evening  of  the  first  day  the  team  visited  and  interviewed  two  individuals 
in  their  apartments.  At  the  end  of  the  first  day,  the  team  members  met  to  discuss 
how  the  review  had  gone,  thus  far,  and  what  adjustments  needed  to  be  made  to 
the  schedule  for  the  remaining  days  of  the  visit. 

Day  Two 

In  the  morning  the  team  members  met  with  the  agency  management  staff  to 
touch  base  about  how  the  logistics  of  the  review  were  working.  After  minor 
adjustments  were  made,  the  individual  reviews  continued.  Observations  were 
made  of  the  day  program  at  the  agency  in  which  many  of  the  individuals 
supported  by  the  agency  are  involved. 

Follow-up  surveys  were  conducted  for  four  individuals.  The  purpose  of  follow- 
up  surveys  is:  "to  determine  if  the  normal  procedures  we  use  for  the  review 
identify  the  critical  issues  for  people  with  regard  to  HEALTH,  SAFETY,  and 
ABUSE  AND  NEGLECT,  and  if  supports  a**i  services  address  the  issues."  The 
follow-up  was  done  after  the  decisions  had  been  made  by  the  team  and  were 
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done  by  the  person  who  did  not  do  the  initial  interview,  to  maintain  the 
objective  nature  of  this  part  of  the  review.  The  follow-up  for  medical  issues  was 
done  using  the  individuals'  records  and  included  the  plan,  any  medical 
evaluation  reports,  consultants'  reports  and  any  other  relevant  medical 
information.  Safety  was  addressed  using  the  individuals'  plans,  evacuation  drill 
reports  and  self-assessments  done  by  the  agency. 

That  evening  visits  were  made  to  the  two  group  homes  and  the  ICF/MR.  The 
visits  were  made  after  individuals  had  returned  home  from  work  or  day 
program,  and  in  same  cases,  just  before  dinner.  Interviews  were  conducted  in 
the  privacy  of  individuals'  rooms  when  that  was  requested.  Later  in  the  evening 
the  team  members  met  to  discuss  the  interviews  and  visits  that  had  taken  place 
that  day.  Final  plans  were  made  to  complete  the  interviews  and  the 
Performance  Measures  for  Organizations  on  Day  Three. 

Day  Three 

The  team  once  again  split  up,  and  while  one  team  member  completed  the 
remaining  individual  reviews,  the  other  team  member  met  with  relevant  staff  to 
complete  the  Performance  Measures  for  Organizations.  Once  all  interviews  were 
completed,  both  with  individuals  and  staff,  the  team  returned  to  their  hotel  to 
begin  the  decision-makmg  process.  The  reviewers  discussed  their  findings  on  a 
daily  basis,  and  in  many  cases  assessed  the  presence  of  the  outcomes  for 
individuals  as  they  had  enough  information  to  do  so.  The  reviewers  used  the 
Outcome  Decision  Making  Guidelines  for  making  their  decisions.  For  each 
individual  a  decision  was  made  of  whether  the  outcome  was  present.  In 
addition,  a  judgment  was  made  of  whether  the  organization  had  a  process  in 
place  to  support  the  individual  in  achieving  the  outcome.  As  the  reviewers 
made  decisions  for  each  individual,  they  noted  specific  areas  where  the  agency 
deserved  commendation  or  needed  improvement. 

Once  all  of  the  assessments  are  made,  the  scores  are  recorded  for  each 
individual.  The  scores  are  totaled  for  each  individual  and  then  across 
individuals  to  develop  an  organizational  profile.  The  Council  has  developed  a 
formula  for  determining  a  recommendation  for  accreditation  status  based  on  the 
average  number  of  outcomes  present  across  individuals  in  the  sample.  Once  a 
decision  is  made,  a  recommendation  is  presented  to  the  Accreditation 
Committee  of  the  Council  for  their  consideration  and  final  determination. 

This  review  was  particularly  interesting,  in  that  the  decision-making  process  of 
the  team  resulted  in  a  recommendation  that  the  agency  not  receive  accreditation. 
This  was  contrary  to  the  results  of  the  self-assessment  performed  by  the  agency. 
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Day  Four 


On  the  last  day  of  the  survey,  the  reviewers  are  provided  with  an  opportunity  to 
present  the  information  to  the  staff.  Typically  a  preliminary  meeting  is  held 
with  a  small  group  of  key  staff  and  then  a  meeting  is  held  with  a  larger  group  to 
share  the  findings  of  the  survey.  Because  of  the  results  of  this  survey,  it  was 
decided  that  a  meeting  would  be  held  first  with  the  Executive  Director,  leaving 
the  decisions  about  subsequent  meetings  to  him.  The  Executive  Director  decided 
that  the  subsequent  meeting  should  be  restricted  to  key  staff,  i.e.,  primarily  those 
staff  involved  in  the  initial  meeting. 

Impressions  and  Issues 

Having  been  only  vaguely  familiar  with  The  Accreditation  Council  process,  this 
review  provided  the  Temple  observer  with  an  excellent  opportunity  to  observe 
the  application  of  the  new,  outcome  standards.  There  is  no  question  that  the 
publication  of  these  standards  represents  a  significant  shift  from  the  process- 
oriented  standards  that  characterized  ACDD  for  so  many  years.  The  outcome 
assessment  process  is  invaluable  in  focusing  the  field  on  the  purpose  of  service 
delivery  to  people  with  developmental  disabilities  -  the  production  of 
measurable  outcomes.  The  outcomes  measured  by  the  accreditation  process 
closely  parallel  those  developed  by  the  HCFA  working  group  in  November, 
1993  and  refined  by  the  Technical  Working  Group  to  this  project  in  November, 
1994.  Those  outcomes  include: 


The  survey  process  provides  an  opportunity  for  organizational  growth  and 
development,  and  the  reviewers  were  able  to  provide  ongoing  technical 
assistance  to  the  agency  throughout  the  process. 

However,  there  are  still  some  concerns  remaining  regarding  the  Accreditation 
process.  Some  of  the  issues  raised  have  been  observed;  others  have  been 
expressed  by  colleagues  in  the  field.  First,  there  are  some  concerns  about  the 
sampling  process.  It  has  been  made  very  clear  from  the  beginning  of  the 
promulgation  of  the  Outcome  Standards  that  the  individuals  determine  what 
outcomes  are  important  to  them.  If  this  is  the  case,  how  can  any  one  assume  that 
the  outcomes  that  are  important  to  one  person  are  generalizable  to  others?  There 


Self-determination 
Dignity 

Health  and  Safety 
Consumer  Satisfaction 


Individualism 
Personal  Growth 
Community  Inclusion 
Relationships/ Social 


Connections 
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are  also  some  concerns  regarding  the  numbers  of  individuals  in  the  sample. 
Although  the  need  for  making  efficient  use  of  time  is  understood,  the 
stratification  that  is  desired  (as  reflected  in  the  Outcome  Based  Performance 
Measures  Manual,  page  32)  is  hardly  achievable  from  a  truly  scientific 
perspective.  Sampling  one  individual  from  each  of  the  group  homes,  one 
individual  from  the  ICF/MR,  one  individual  receiving  individual  support 
services  and  three  individuals  from  the  apartment  programs  did  not  seem  to 
accurately  reflect  the  breadth  of  individuals  supported  by  this  agency.  Although 
there  is  no  immediate  suggestion  for  remediation,  short  of  increasing  the  sample 
size,  this  is  noted  as  an  issue. 

Secondly,  the  issue  is  raised  of  who  conducts  the  AC  surveys.  Although  the 
Temple  observer  is  not  sure  of  the  particular  requirements  for  surveyors  other 
than  years  of  demonstrated  experience  providing  direct  support  to  individuals 
with  developmental  disabilities,  there  is  no  requirement  that  individuals  work 
or  have  experience  working  in  accredited  facilities.  This  may  be  a  small  issue; 
however,  it  has  been  raised  as  a  cause  for  concern,  particularly  in  the  surveyors' 
role  as  providers  of  technical  assistance. 

Third,  the  issue  of  health  and  safety  is  raised.  There  are  specific  outcomes  that 
address  the  issues  of  health  and  safety  (outcomes  16,  21,  22  and  23).  In  addition 
to  the  survey  process,  AC  has  focused  additional  attention  on  those  areas  for  a 
subsample  of  individuals.  However,  can  AC  truly  profess  that  in  an  accredited 
agency  all  individuals  are  healthy,  safe  and  free  from  abuse  and  neglect?  Can 
any  system  make  that  claim?  The  question,  then,  is  can  the  accreditation  process 
be  modified  in  such  a  way  to  include  more  individuals,  such  that  the  issues  of 
health  and  safety  are  directly  addressed?  For  example,  one  strategy  may  be  to 
expand  the  sample  to  include  those  individuals  for  whom  there  are  behavior 
programs,  high  numbers  of  unusual  incident  reports  or  medical  fragilities.  This 
may  better  address  the  health  and  safety  issues. 

Fourth,  is  the  issue  of  applicability  across  settings.  HCFA  is  concerned  that  any 
system  be  applicable  in  all  settings,  including  those  supporting  people  with  the 
most  significant  disabilities,  including  those  who  are  non-ambulatory  and  those 
individuals  who  experience  speech  and  communication  difficulties.  To  the  best 
of  the  research  team's  knowledge,  thus  far,  no  agency  supporting  more  than  120 
individuals  in  one  setting  has  received  accreditation  under  the  outcome 
standards.  The  remaining  question  is  whether  any  large  facility  can  receive 
accreditation. 

Fifth,  although  the  Accreditation  Council  is  vigorously  pursuing  the  question  of 
inter-rater  reliability,  there  have  been  no  studies  to  date  of  the  concurrent 
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validity  of  the  instrument.  It  is  our  understanding  that  this  will  be  pursued  in 
the  future. 

Description  of  Data  System 

AC  data  are  collected  in  a  way  that  is  amenable  to  some  levels  of  analysis.  Each 
individual  in  the  sample  is  given  a  score,  based  on  the  number  of  outcomes 
present  and  the  presence  of  a  process  in  place  to  support  the  outcomes.  The 
scores  for  individuals  are  summarized  to  obtain  a  total  score  for  the  agency. 

Reliability 

The  Accreditation  Council  has  been  actively  involved  in  assessing  the  reliability 
of  the  outcome  standards.  As  mentioned  previously,  all  reviews  include  a 
reliability  assessment.  AC  is  also  involved  in  a  more  comprehensive  assessment 
of  reliability.  Once  the  results  of  the  study  are  made  available,  they  will  be 
forwarded  to  HCFA.  Because  AC  is  in  the  process  of  a  rigorous  reliability  study, 
no  further  work  in  this  area  was  undertaken  by  the  study  team. 

Concurrent  Validity 

The  Accreditation  Council  is  very  interested  in  examining  the  concurrent 
validity  of  the  outcome  measures,  however,  that  assessment  has  yet  to  be 
completed.  As  a  result,  the  Temple  team  has  received  from  AC  a  listing  of  those 
agencies  across  the  country  who  have  been  accredited  utilizing  the  outcome 
measures.  After  reviewing  the  list,  an  agency  in  Pennsylvania  was  solicited  for 
their  participation  in  this  phase  of  the  study.  A  data  collector  went  to  that 
facility  and  collected  information  about  30  individuals,  using  the  Temple  BDS. 
The  30  individuals  were  selected  by  the  agency  as  a  sample  of  convenience.  For 
reasons  of  confidentiality  it  was  not  disclosed  whether  any  of  the  individuals  in 
this  portion  of  the  study  also  participated  in  the  Accreditation  Survey  the  year 
before.  The  agency  made  available  to  Temple  a  full  copy  of  the  Accreditation 
Report,  which  was  issued  in  March,  1995.  The  first  caution  here  is  that  the  data 
collected  were  collected  approximately  one  year  after  the  Accreditation  Survey. 
There  is  no  doubt  that  significant  change  occurred  during  that  one  year  period. 

Once  the  report  was  received,  the  study  team  spent  considerable  time  discussing 
how  the  AC  data  could  be  best  used  for  purposes  of  assessing  concurrent 
validity.  The  first  step  in  this  process  was  to  categorize  the  30  AC  standards  into 
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the  valued  outcomes  developed  by  the  HCFA  working  group.  The 
categorization  is  as  follows: 


HCFA  OUTCOMES 


AC  OUTCOMES 


Self-Determination 


Personal  Growth 
Integration 


Dignity 


Relationships/Social  Connections 


Health  and  Safety 


Individualization 


People  choose  personal  goals. 
People  choose  where  and  with  whom 
they  live. 

People  choose  where  they  work. 
People  decide  how  to  use  their  free 
time. 

People  choose  services. 

People  choose  their  daily  routine. 

People  decide  when  to  share  personal 

information. 

People  realize  personal  goals. 
People  participate  in  the  life  of  the 
community. 

People  interact  with  other  members  of 
the  community. 

People  live  in  integrated  environments 

People  perform  different  social  roles. 

People  exercise  rights. 

People  are  respected. 

People  have  economic  resources. 

People  have  friends. 

People  remain  connected  to  natural 

support  networks. 

People  have  intimate  relationships. 

People  are  afforded  due  process  if 

rights  are  limited. 

People  are  free  from  abuse  &  neglect. 
People  have  health  care  services. 
People  have  the  best  possible  health. 
People  are  safe. 

People  have  insurance  to  protect  their 
resources. 

People  experience  continuity 
and  security. 

People  have  time,  space  and 
opportunity  for  privacy. 
People  have  and  keep  personal 
possessions. 

People  use  their  environments. 
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Satisfaction  People  are  satisfied  with  services. 

People  are  satisfied  with  their 
personal  life  situations. 

Once  the  AC  data  were  categorized  into  the  outcomes  identified  by  the  HCFA 
working  group,  the  study  team  began  to  analyze  the  data.  In  the  AC  process,  for 
each  individual  outcome,  a  determination  is  made  as  to  how  many  individuals 
in  the  sample  achieved  the  outcome.  Additionally,  a  rating  is  given  to  the 
number  of  processes  in  place  to  support  the  achievement  of  the  outcome.  For 
this  study,  the  team  calculated  the  scores  for  each  of  the  HCFA  outcomes.  In 
order  to  do  this,  for  each  item  in  the  outcome  the  number  of  outcomes  were 
summed  with  the  number  of  processes  present.  Those  sum  scores  were  then 
divided  by  the  total  number  of  items  comprising  the  outcome,  to  produce  a 
percentage  score  against  which  other  measures  (in  this  case  the  Temple  BDS) 
could  be  compared.  The  analysis  yielded  the  following  information: 

Outcome  AC  Percent  Temple  Percent 

Individualization  77  28 

Health  &  Safety  84  93 

Satisfaction  82  54 

Personal  Growth  73 

Relationships  64 

Integration  43  20 

Self-Determination  55  27 

Dignity  49  97 

The  two  areas  where  there  is  the  greatest  correspondence  are  the  areas  of  health 
and  safety  and  integration.  In  the  other  areas,  there  is  little  to  no  relationship. 
These  data  must  be  interpreted  with  extreme  caution.  As  mentioned  previously, 
the  two  types  of  data  were  collected  one  year  apart.  Second,  the  samples  were 
both  purposive,  but  in  different  ways.  The  AC  survey  attempted  to  include 
individuals  from  a  variety  of  settings  (7  people  from  community  settings,  5  from 
the  large  ICF/MR  and  3  who  receive  day  services  only);  the  Temple  sample  was 
originally  to  include  equal  proportions  of  individuals  in  community  and 
residential  settings.  However,  as  data  were  being  collected  a  snowstorm 
arrived,  necessitating  more  data  collection  in  the  large  ICF/MR  where  staff  were 
available  for  interviews.  The  result  of  collecting  2/3  of  the  data  in  the  large 
ICF/MR  setting  is  reflected  in  particular,  in  the  dimensions  of  individualization, 
self-determination  and  dignity.  Third,  the  AC  process  actually  yields  a  facility 
rating  rather  than  individual  ratings.  As  a  result,  the  one  AC  score  is  compared 
to  30  Temple  BDS  scores;  hence  no  correlations  were  possible. 
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In  summary,  we  encourage  the  Accreditation  Council  to  continue  its  pursuit  of 
the  assessment  of  concurrent  validity  of  the  Accreditation  process.  The  study 
team  would  be  pleased  to  collaborate  with  the  Accreditation  Council  in  this 
effort 

Applicability 

As  mentioned  previously,  thus  far  no  large  agency  or  facility  (larger  than  120 
individuals)  has  been  accredited  using  the  outcome  standards.  Because  of  the 
intensity  of  the  accreditation  process  and  the  need  for  highly  trained  surveyors, 
it  is  impractical  for  the  Temple  team  to  conduct  an  AC  assessment.  At  this  point 
it  is  questionable  as  to  whether  these  standards  can  be  applied  successfully  in  a 
large  setting. 

Discriminant  Validity 

The  Accreditation  Council  does  not  maintain  quantitative  scores  for  agencies. 
Rather  accreditation  is  given  for  a  one  or  two  year  period.  In  addition,  a 
decision  is  made  for  an  entire  agency.  Therefore,  the  data  emanating  from  AC 
surveys  are  not  amenable  to  this  type  of  analysis. 
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CARF  SURVEY  REPORT 


Overview 

The  purpose  of  this  visit  was  to  observe  the  implementation  of  the  new  (1995) 
standards  for  community  support  accreditation  from  CARF.  These  new  standards 
were  developed  by  national  consensus  and  represent  a  major  change  in  the  method  of 
review  and  the  measures  of  quality  for  the  CARF  survey.  Previously,  the  review  was 
based  primarily  in  documentation  review  in  order  to  establish  compliance  with 
standards.  The  revised  standards  are  based  in  interview  with  staff  and  the 
service /support  recipients  using  documentation  as  the  source  of  proof  for  planning, 
involvement  of  consumers  and  family  in  choice  and  decision-making,  staff  training  and 
health  and  safety  issues.  The  CARF  survey  in  Minnetonka,  Minnesota  occurred 
between  July  24  -  26,  1995.  During  the  three  day  visit,  Robin  Levine,  a  Temple  staff 
person  was  part  of  all  planning  and  discussion  meetings  and  accompanied  the 
residential  reviewer  to  all  agency  residential  sites. 

The  Survey  Standards,  Structure  and  Process 

The  CARF  survey  is  a  peer  review  conducted  on  site  by  a  team  of  reviewers.  The 
number  of  reviewers  and  days  for  the  review  are  determined  by  CARF  staff  based  on 
the  size  of  the  organization  and  the  number  of  programs  to  be  reviewed.  The  site 
review  team  was  comprised  of  three  reviewers,  each  responsible  for  one  area  of 
program  (administration,  residential  and  vocational  supports).  CARF  reviewers  are 
required  to  work  in  CARF  accredited  agencies  and  hold  positions  in  direct 
correspondence  to  their  area  of  review.  Including  people  with  disabilities  and  family 
members  as  reviewers  is  being  considered  by  the  CARF  Board  of  Directors. 

All  CARF  surveyors  are  required  to  perform  a  minimum  of  three  surveys  per  year. 
This  is  set  forth  due  to  the  amount  of  time  and  resources  expended  to  train  each 
reviewer.  Training  of  new  reviewers  takes  pake  over  five  days  at  the  CARF  National 
Office  in  Tuscon,  Arizona.  The  training  is  paid  for  by  CARF  and  generally  includes  20 
to  30  people  in  the  trianing  group.  The  training  includes  a  thorough  didactic  review  of 
the  standards  followed  by  a  two  day  expereintial  role  play  of  the  team  dynamic  and 
survey  process.  This  two  day  role  play  includes  writing  examples,  introduction  and 
exit  meetings,  communication  techniques  and  active  listening.  Trainees  receive 
feedback  on  their  performance  throughout  the  training.  At  the  end  of  the  five  days  if 
the  trainee  is  accepted  as  a  reviewer,  he/she  becomes  an  intern  with  a  seasoned 
surveyor  identified  as  a  mentor.  Over  the  next  three  to  six  months  the  Intern 
participates  in  two  to  three  survey*  as  a  regular  team  member.   The  perfomrance 
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during  each  review  is  critiqued  by  the  mentor,  other  team  mebers  and  the  organization. 
At  the  end  of  the  internship  period,  the  reviewer  can  graduate  (becoming  an  approved 
CARF  surveyor),  receive  counseling  toward  improved  performance  of  continue  in  the 
internship  program.  Additional  training  has  been  required  since  1994,  after  the  new 
standards  were  adopted  by  CARF.  This  continuing  education  is  provdide  in  Tuscon 
and  at  other  regional  locations  by  CARF  staff.  All  surveyors  are  required  to  attend  one 
contining  education  training  every  three  years. 

The  three  reviewers  at  this  review  were  experienced  CARF  surveyors,  each  having 
participated  in  more  than  30  reviews  over  the  past  years.  The  administration  reviewer 
is  identified  as  the  team  leader  and  takes  responsibility  for  overall  scheduling,  team 
meetings  and  final  report  writing.  This  was  the  first  review  for  all  the  reviewers 
utilizing  the  new  standards.  All  expressed  initial  concern  about  effective 
implementation  of  the  new  standards  but  expressed  their  overall  belief  in  the  strength 
and  applicability  of  the  CARF  survey. 

The  survey  standards  are  based  on  four  overarching  accreditation  principles  and  eight 
accreditation  criteria  (this  is  the  substance  of  the  administrative  review).  The  eight 
accreditation  criteria  must  have  written  policies  to  document  conformance  with  the 
standards  and  these  policies  are  submitted  to  the  review  team  in  advance  of  the  review. 
The  eight  policies  are:  1)  input  from  the  persons  served;  2)  accessibility;  3)  outcomes;  4) 
rights,  health  and  safety;  5)  human  resource  development;  6)  leadership;  7)  legal 
requirements;  and  8)  financial  planning  and  management.  If  one  or  more  of  these 
policies  do  not  exist,  the  agency  accreditation  is  placed  on  a  60  day  hold  until 
conformance  is  achieved.  The  survey  is  designed  to  measure  quality  outcomes  in  three 
identified  core  areas.  The  core  areas  to  be  reviewed  during  this  survey  were 
Promoting  Organizational  Quality  (administration),  Promoting  Service  Quality 
(consumer  satisfaction  and  service  outcomes)  and  Specific  Services/Program  Standards 
(residential,  employment,  personal  and  social  issues).  All  agencies  must  be  reviewed 
for  organizational  quality  regardless  of  which  service /support  program  is  being 
specifically  accredited.  Each  core  area  has  a  set  or  several  sets  of  standards  (criteria) 
with  interpretive  guidelines.  The  standards  for  Promoting  Organizational  Quality  have 
been  described  in  the  eight  policies  identified  above.  Promoting  service  quality 
addresses  two  standards:  1)  individual-centered  planning,  and  2)  service  design  and 
delivery.  Specific  Services /Program  Standards  identify  14  standards  in  the  area  of 
community  support  services.  The  first  9  of  14  standards  address  supports  to  the  person 
served  in  the  areas  of:  1)  tailored  support  services,  2)  assistive  technology  and 
reasonable  accommodation;  3)  support  in  decisionmaking;  4)  informed  choice;  5) 
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learning  new  skills;  6)  self-sufficiency;  7)  relationships;  8)  resolution  of  conflicts;  and  9) 
risk  to  health  and  safety.  Five  of  the  14  standards  measure  organizational  support  to 
the  person  in  their  ability  to  1)  demonstrate  positive  changes  (data  collection);  2)  assist 
in  maintenance  of  assistive  technology  and  equipment;  3)  provide  services  based  on  the 
concept  of  natural  proportions;  4)  promote  community  inclusion;  and  5)  develop  or 
facilitate  access  to  community  supports. 

The  task  of  the  review  team  is  to  determine  conformance  or  non  conformance  with  the 
standards  based  on  their  interviews,  record  review,  observations  and  judgment. 
Ratings  of  conformance  can  be  accompanied  by  a  commendation  for  best  practice.  A 
rating  of  non-conformance  is  given  with  either  a  suggestion  for  improvement  (does  not 
require  follow-up)  or  a  recommendation  (requires  a  90  day  corrective  action  plan).  The 
corrective  action  plan  is  compared  to  a  quality  improvement  plan.  The  team  writes  a 
report  of  their  findings  prior  to  leaving  the  site  visit.  This  report  is  shared  with  the 
management  staff  of  the  organization  during  the  exit  interview.  It  is  customary  for  the 
Executive  Director  to  have  a  private  exit  interview  in  advance  of  the  larger  staff  exit 
interview.  The  site  reviewers'  report  is  submitted  to  the  CARF  national  office  to  be 
revised  into  "CARFese"  by  editors  .  This  is  done  in  an  effort  to  bring  consistency  in 
form  and  language  to  all  reports.  This  revised  report  is  then  submitted  to  the  CARF 
Board  of  Directors  who  is  the  accreditation  granting  body.  The  team  makes  a 
recommendation  of  a  non,  one,  or  three  year  accreditation  to  the  Board  but  it  is  the 
Board  who  makes  the  final  decision. 

Documentation  and  Observation  of  the  Site  Review  Process 

In  advance  of  the  on-site  review,  the  agency  completed  a  self-study  questionnaire 
which  was  distributed  to  the  site  review  team  along  with  general  agency  brochures. 

Day  One 

The  team  held  an  introductory  meeting  for  all  key  management  and  program  staff  on 
the  first  morning  of  the  review.  This  meeting  included  the  process  and  goals  of  the 
review,  introduction  of  the  reviewers  and  Temple  observer,  introduction  of  all  agency 
staff  at  the  meeting  and  a  presentation  from  the  Executive  Director  on  the  agency 
history,  mission,  goals  and  objectives. 

After  this  introductory  meeting  the  three  reviewers  were  given  individual  schedules 
prepared  by  the  agency  which  would  take  them  to  all  the  program  sites  of  this  agency. 
Agency  staff  accompanied  the  reviewers  to  each  location  allowing  for  individual  staff 
interviews  during  the  driving  time. 


3 


At  the  end  of  Day  1,  the  reviewers  met  back  at  the  hotel  in  the  evening  for  a  debriefing. 
At  this  time  the  team  leader  checked  in  on  scheduling  needs  and  emerging  problems 
were  identified.  In  this  case,  and  much  to  the  surprise  of  the  team,  this  agency 
exhibited  serious  administrative,  program  and  health  and  safety  issues.  The  most 
significant  issue,  both  in  terms  of  overall  health,  safety  and  quality  of  life  for  people, 
was  a  100-bed  facility  that  the  team  had  dinner  at  that  evening.  The  facility  included 
50  one-person  rooms  and  50  two-person  rooms  of  equal  size.  In  touring  the  facility  the 
agency  staff  were  working  hard  to  compare  this  facility  to  a  college  dorm  and  to 
highlight  all  the  choices  and  satisfaction  that  the  people  were  experiencing.  What  was 
observed  was  an  outdated  facility  with  poor  ventilation,  heating  and  cooling,  cramped 
rooms,  outdated  electrical  fittings  and  a  pervasive  smell  of  urine.  All  the  reviewers 
expressed  their  dismay  with  this  facility  and  were  prepared  to  identify  the  specific 
standards  that  would  support  a  finding  of  non-conformance  in  several  core  area 
standards.  It  was  agreed  that  the  team  leader  would  begin  to  give  this  information  to 
the  Executive  Director  in  an  effort  to  prepare  him  for  the  final  report  which  would 
include  many  recommendations  requiring  the  90  day  plan  of  correction. 

Day  Two 

The  site  review  team  members  followed  their  individual  schedules  of  site  visits  and 
interviews  for  the  day  without  conferring  with  one  another.  The  residential  reviewer 
returned  to  the  large  facility  to  review  records  in  addition  to  seeing  several  other  group 
homes  and  supported  living  sites.  In  all  sites  he  talked  to  people  about  their  choices 
and  satisfaction  and  then  looked  for  documentation  to  corroborate  their  involvement. 
In  each  residence  fire  safety,  medications  administration  and  emergency  evacuation 
procedures  were  investigated.  Unfortunately,  the  reviewer  did  not  always  meet  the 
people  who  lived  in  these  homes  due  to  limited  time.  It  is  not  required  that  the 
reviewer  meet  people  in  every  location. 

At  the  end  of  Day  2  the  team  met  again  to  discuss  remaining  programs  to  be  visited 
and  to  build  consensus  on  the  overall  evaluation  of  this  agency.  The  responsible  team 
member  went  through  each  standard  and  guideline  and  described  the  conformance  or 
non  conformance  rating.  This  seemed  to  be  a  particularly  beneficial  process  in  that 
new  information  was  shared  among  site  reviewers.  Interestingly,  the  review  team 
members  said  that  this  was  the  first  time  they  had  ever  done  this  type  of  reconciliation. 
Perhaps  it  was  needed  due  to  the  new  standards  and  process  or  because  of  the  serious 
problems  identified  within  this  agency.  After  the  discussion  of  individual  ratings  each 
team  member  made  their  recommendation  for  number  of  years  of  accreditation. 
Following  this  meeting  the  reviewers  began  to  write  their  individual  reports  to  be 
finished  and  reported  in  summary  on  Day  3. 
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Day  Three 


The  morning  was  spent  finishing  up  site  visits  and  collecting  documentation  that  was 
found  to  be  missing  during  report  writing.  Final  reports  were  written  and  submitted  to 
the  team  leader. 

A  pre-exit  interview  giving  a  brief  description  of  the  findings  was  held  with  the 
Executive  Director  prior  to  the  staff  exit  interview.  At  both  meetings  the  reviewers 
were  very  purposeful  in  speaking  as  "we  the  team"  and  framing  the  CARF  survey  as  an 
opportunity  for  growth  and  improvement.  Each  reviewer  read  their  report  in  its 
entirety  with  added  extemporaneous  comments.  Site  reviewers  were  available  for  a 
short  time  after  the  exit  interview  to  discuss  specific  findings  and  accept  additional 
documentation. 

Description  of  Data  System 

As  described  in  the  summary  of  the  site  visit,  CARF  surveys  do  not  result  in  a 
quantification  of  ratings.  The  ratings  of  individual  site  reviewers  for  each  measure  is 
maintained  by  the  reviewer  if  additional  information  is  requested  by  the  CARF  Board 
of  Directors.  The  report  from  the  site  review  team  is  written  in  narrative  style.  Specific 
measures  are  identified  only  in  the  case  of  a  recommendation  or  commendation.  The 
measures  that  are  not  specifically  identified  are  included  in  a  summary  overview  and 
rating  of  each  section. 

Reliability 

Reliability  of  the  CARF  review  was  not  conducted  by  Temple  because  of  two  factors. 
First,  as  previously  described,  the  CARF  survey  process  is  conducted  by  a  team  of  peer 
site  reviewers  who  have  undergone  extensive  training  and  are  working  in  CARF 
accredited  agencies.  From  a  research  perspective,  it  would  be  inappropriate  to  have 
Temple  data  collectors  assume  the  role  of  a  CARF  reviewer  without  the  same  level  and 
degree  of  training  and  without  the  experience  of  working  in  a  CARF  accredited  facility. 
Additionally,  a  CARF  review  takes  many  days  to  complete  and  requires  the  total 
commitment  of  the  agency  under  review.  It  would  not  be  within  acceptable  research 
protocol  to  request  that  an  agency  undergo  a  second  CARF  survey  for  purposes  of 
establishing  reliability  of  the  instrument.  It  would  be  worthwhile,  however,  as  CARF 
continues  to  utilize  the  outcome  standards  to  undertake  such  an  effort. 
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Concurrent  Validity 


In  order  to  demonstrate  concurrent  validity,  the  extent  to  which  the  CARF  instrument 
measures  the  same  underlying  dimensions  as  other  instruments  with  established 
statistical  properties,  CARF  data  were  compared  to  the  Temple  University  Behavior 
Development  Survey  (BDS).  In  order  to  accomplish  this,  a  quantification  of  the  CARF 
ratings  and  the  collection  of  the  Temple  BDS  in  a  CARF  facility  accredited  by  the  1995 
outcome  based  standards  was  conducted. 


With  the  assistance  of  administrative  staff  at  CARF,  an  agency  was  identified  that  had 
been  reviewed  using  the  1995  outcome  based  standards.  This  agency  was  approached 
by  Temple  staff  and  agreed  to  participate  in  the  concurrent  validity  analysis.  The 
agency  also  agreed  to  provide  us  with  the  actual  CARF  report  from  which  we  would 
compare  findings.  In  March  1996,  a  Temple  data  collector  spent  three  days  in  this 
facility  which  provided  community  living  services  and  supports  to  people  with  varying 
degrees  of  mental  retardation.  Data  were  collected  in  state-funded  group  homes  and  in 
federally  funded  ICF/MR  programs.  Thirty  Temple  University  BDSs  were  collected 
about  individuals  in  this  agency  who  were  residing  in  the  agency  at  the  time  of  the 
CARF  review. 


In  order  to  compare  the  six  domains  of  the  Temple  University  BDS  which  addressed 
the  valued  outcomes  identified  by  the  HCFA  Working  Group,  CARF  standards  have 
been  classified  into  the  following  comparable  Temple  BDS  domains. 

Domain  of  Temple  BDS  Comparable  CARF  Standards 


Health  and  Safety 


Individualization 


Accommodation 


IB  -  Accessibility 

ID  -  Rights,  Health  and  Safety 

3B.9  -  Assessment  of  Risk  to  Health  and  Safety 

3B.32  -  Individual  Health  Care  and  Safety 

1 A  -  Input  of  People  Served 
2A  -  Individual-Centered  Planning 
3B.1  -  Individually  Tailored  Supports 
3B.2  -  Assistive  Technology/Reasonable 

3B.3  -  Decision  Making 

3B.4  -  Informed  Choice 

3B.11  -Appropriate  Assistive  Technology 

3B.30  -Personal  Space  and  Privacy 

3B.31  -  Alternate  Living  Opportunities 
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Integration/Inclusion 


3B.12  -  Natural  Proportions  in  the  Community 
3B.13  -  Promotes  and  Facilitates  Inclusion 
3B.14  -  Access  to  Community  Supports 


Relationships/Socialization 


3B.7  -  Meaningful  Relationships 
3B.8  -  Resolution  of  Conflicts 


Personal  Growth 


3B.5  -  Learn  New  Skills 
3B.6  -  Self  -  Sufficiency 


Consumer  Satisfaction 


1C.7  -  Use  of  Consumer  Satisfaction  Information 
2B  -Evaluation  of  Consumer  Satisfaction  and 
Outcomes 

3B.10  -Demonstrates  Positive  Changes 


The  most  complex  task  in  this  analysis  was  actually  quantifying  the  CARF  review.  The 
results  of  the  review  are  provided  in  a  narrative  format.  There  are  no  raw  scores 
provided.  Each  standard  is  discussed  in  paragraph  form  but  each  criterion  within  the 
standard  is  not  necessarily  noted  or  referenced.  A  final  rating  for  each  standard  is  not 
explicitly  noted  while  rating  language  is  implicit  throughout  the  narrative  report.  For 
purposes  of  this  analysis  the  narrative  report  was  thoroughly  reviewed  by  Temple  staff. 
Within  each  category  ratings  were  given  if  an  explicit  recommendation  or 
commendation  was  specifically  tagged  to  an  individual  standard.  If  a  specific 
recommendation  or  commendation  was  not  noted  it  was  determined  that  the  agency 
had  conformed  with  the  standard  and  the  corresponding  rating  was  attributed.  For 
purposes  of  comparison  to  the  score  in  each  of  the  six  domains  of  the  Temple  BDS  the 
following  numeric  rating  scale  has  been  developed  for  the  CARF  ratings. 

Conformance  with  Commendation  =  4 
Conformance  =  3 
Conformance  with  Suggestion  =  2 
Non-Conformance  with  Recommendation  =  1 

In  the  analysis  of  this  data,  we  calculated  the  sum  scores  of  the  ten  Temple  BDS 
domains  from  BDSs  collected  at  the  CARF  accredited  agency.  We  also  computed  the 
sum  scores  of  the  seven  Health  and  Safety  scales  also  collected  at  the  CARF  agency.  For 
the  ratings  attributed  to  the  original  CARF  survey,  we  calculated  the  mean  score  and 
percentile  ratings  for  the  six  categories  identified.  The  table  below  displays  the 
preliminary  findings  of  this  analysis. 

HCFA  Work  Group  Outcome  Domains         Percentile  (CARF  Survey) 


Individualization 
Health  and  Safety 
Satisfaction 


80 
74 
73 


Personal  Growth 

Relationships 

Integration 


67 
50 
67 


Temple  BDS  Domains 


Percentile  (Temple  BDS) 


Dignity 

Adaptive  Behavior 

Challenging  Behavior  Frequency 

Challenging  Behavior  Severity 

Frequency  of  Integrative  Activities 

Consumer  Satisfaction 

Individualization 

Self  Determination 

Medications 

Med  Needs 

Behavior  Plan 

Safety  Scale 

Safety  of  Neighborhood 


100 
73 
77 
87 
38 
62 
62 
44 
70 
99 
88 
96 
76 


Health  and  Safety  Mean  Score 


86 


There  are  four  categories  of  the  HCFA  domains  that  can  be  compared  to  the  Temple 
BDS  domains.  Health  and  Safety  in  the  CARF  survey  (74)  compared  to  the  Health  and 
Safety  Mean  Score  from  the  Temple  BDS  (86).  Individualization  on  the  CARF  survey 
yielded  a  score  of  80,  compared  to  a  score  of  62  on  the  Temple  BDS.  The  Satisfaction 
measure  on  the  CARF  survey  was  73,  while  the  Temple  BDS  score  was  62.  Integration 
on  the  CARF  survey  was  67,  with  a  score  of  38  on  the  Temple  BDS.  Integration  scores 
produced  the  most  disparate  findings. 

While  it  took  considerable  effort  to  match  CARF  standards  to  Temple  BDS  items,  it 
appears  that  there  is  some  level  of  correspondence  between  the  outcomes  of  health  and 
safety  and  consumer  satisfaction.  However,  it  bears  repeating  that  the  CARF  survey 
process  does  not  yield  quantitative  outcome  measures. 

Applicability 

The  CARF  review  process  is  conducted  in  agencies  that  offer  a  variety  of  living 
supports  to  people  with  varying  degrees  of  mental  retardation  including  group  homes 
and  ICF/MR  facilities.  The  CARF  review  has  been  found  to  be  applicable  to  these 
settings. 
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Discriminant  Validity 


Due  to  the  fact  that  CARF  ratings  are  not  quantified  and  do  not  generate  numeric  raw 
scores,  a  discriminant  validity  analysis  is  unable  to  be  conducted. 

It  is  recommended  that  CARF  continue  to  evaluate  the  statistical  properties  of  the 
outcome  based  system. 

Conclusions 

This  process  was  found  to  be  thorough  and  valuable  to  the  agency  from  a  quality 
assurance  and  growth  perspective.  The  reviewers  were  highly  qualified,  comfortable 
and  capable  in  talking  with  people  with  disabilities  and  staff,  and  thorough  in  their 
review  of  materials.  Due  to  their  own  work  experiences  and  national  perspective 
gained  from  conducting  CARF  reviews  they  were  able  to  provide  technical  assistance  to 
staff  throughout  the  visit.  There  is  a  high  level  of  confidence  that  this  review  team  had 
identified  the  health  and  safety  problems  for  the  people  receiving  services /supports 
from  this  agency  and  was  able  to  fairly  measure  the  quality  of  life  opportunities  and 
obstacles  for  these  individuals  as  identified  in  the  CARF  standards. 

There  are  two  areas  of  concern  that  emanate  from  this  review.  Inter-rater  reliability  as 
it  relates  to  the  guarantee  of  quality  and  safety  is  the  first  issue.  It  is  uncertain  how  a 
different  set  of  reviewers  would  have  judged  the  quality  of  these  programs  as  each 
reviewer  brings  a  different  set  of  experiences,  biases  and  preferences  to  the  review. 
While  this  is  addressed  during  training  of  reviewers,  it  is  problematic  that  three  years 
ago  a  site  team  recommended  three  year  accreditation  to  this  agency.  Although  the 
standards  and  process  have  been  greatly  revised,  the  health,  safety  and  quality  issues 
identified  in  the  large  facility  were  present  at  the  time  of  this  previous  review. 

The  second  issue  raised  is  about  the  underlying  premise  in  the  development  of  these 
standards.  Is  this  survey  meant  to  advance  or  reflect  the  field?  This  issue  became  clear 
on  the  first  day  when  the  team  toured  the  sheltered  workshop.  This  was  the  cleanest, 
most  technology  advanced  workshop  the  Temple  observer  had  ever  seen.  People  were 
busy  doing  real  factory  work.  However,  it  was  segregated  from  the  community  and 
even  within  the  workshop  ("the  autistic  program"  was  isolated  into  a  different  area  of 
the  room),  people  were  not  being  paid  minimum  wage  and  goals  were  still  developed 
for  individuals  based  on  a  readiness  model.  On  the  other  hand,  this  agency  has  made 
good  strides  in  supported /supportive  employment  and  will  continue  to  expand  this 
program.  This  agency  has  not  considered  including  people  without  disabilities  in  this 
factory  or  increasing  the  pay  for  workers.  These  kind  of  suggestions  were  not  made  by 
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the  site  review  team  as  this  level  of  quality  and  state  of  the  art  practices  are  not 
explicitly  reflected  in  the  standards  nor  are  they  necessarily  part  of  the  belief  system  of 
these  individual  reviewers. 
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APPENDIX  D 

1       COMMENTS  TO  HCFA  ON  DRAFT  REVISIONS  TO  THE 
SURVEY  PROCESS  FOR  ICFs/MR 

COMMENTS  ON  REVISED  CONDITIONS  OF  PARTICIPATION 

FOR  ICFs/MR 


APPENDIX  D 


COMMENTS  TO  HCFA  ON  DRAFT  REVISIONS 
TO  THE  SURVEY  PROCESS  FOR  ICFs/MR 


General  Comments 

The  approaches  outlined  in  the  Draft  Revisions  appear  to  be  a  clear  improvement  over 
current  survey  procedures.  They  also  appear  to  move  in  the  same  general  direction  of 
the  states  in  quality  assurance  for  residential  services  to  comparable  populations, 
including: 

•  The  use  of  a  smaller  group  of  essential  standards. 

•  The  stronger  focus  on  outcomes. 

•  The  greatly  increased  focus  on  interviews  and  observation,  rather  than 
document  review. 

•  The  retention  of  the  comprehensive  survey  and  its  availability  to  surveyors 
when  problems  are  observed  using  the  Fundamental  Survey  Process. 

There  are  also  some  guidelines  that  appear  to  emphasize  aspects  of  the  ICF/MR 
standards  that  are  particularly  consistent  with  contemporary  values  and  outcomes  that 
are  important  to  individuals  with  mental  retardation/developmental  disabilities,  as  well  as 
the  primary  outcomes  identified  by  the  Technical  Working  Group  advising  the  current 
HSRI  project  on  quality  assurance  in  the  ICF/MR  program: 

•  Individualization  •  Health  and  safety 

•  Self-determination  •  Relationships/social  connections 

•  Dignity  •  Consumer  satisfaction 

•  Integration/inclusion  •  Personal  growth 

Overall,  however,  it  appears  that  the  emphasis  on  individualization,  relationships/social 
connections,  self-determination  and  community  inclusion/integration  is  relatively  weak. 
Although  mentioned  in  some  components  of  the  Draft  Revisions,  these  themes  are  not 
consistently  emphasized  throughout.  Suggestions  on  how  this  emphasis  could  be 
enhanced  are  provided  below. 

As  discussed  in  the  comments  on  Part  IX  of  the  Survey  Process  and  on  the  Suggested 
Interview  Questions,  we  recommend  that  additional  thought  be  given  to  the  use  of 
interviews  with  people  with  communication  limitations  and/or  who  use  different  modes 
of  communication,  as  well  as  those  with  more  severe  levels  of  cognitive  disability.  If 
interviews  are  going  to  become  the  primary  tool  for  surveyors  along  with  observations, 
it  is  important  to  provide  guidance  on  how  to  apply  these  techniques  for  the  many 
thousands  of  ICF/MR  residents  who  will  have  problems  responding  to  the  kinds  of 
questions  that  are  listed. 
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Finally,  we  recommend  that  HCFA  consider  some  use  of  process  indicators,  at  least  in 
the  guidelines  to  surveyors,  specifically  those  procedures  that  are  directly  tied  to  target 
outcomes,  such  as  how  and  what  staff  know  about  individual  communication, 
preferences,  strengths,  choices,  and  interests.  The  continuing  need  to  examine  process 
even  in  an  outcome-focused  survey  is  acknowledged,  for  example,  by  the  Accreditation 
Council  in  the  survey  procedures  for  its  1992  standards.  A  few  suggested  questions  on 
process  that  might  be  included  in  the  guidelines  to  surveyors  are  noted  below  in  relation 
to  specific  sections  of  the  Draft  Revisions. 

Facility  Compliance  Principles 

This  section  provides  good  guidance  overall.  We  recommend  that  it  come  first  and  that 
it  apply  to  ail  surveys,  not  just  Extended  and  Full  Surveys.  The  references  to  choice, 
community  involvement,  and  relationships  in  W122,  and  to  independence, 
communication,  coping,  social  and  leisure  skills  in  W266  are  appropriate,  and  should  be 
mirrored  in  guidance  for  the  Fundamental  Survey  Process. 

The  guidance  on  meeting  the  CoP  of  Active  Treatment  also  should  include  the  concept 
of  choice.  This  is  consistent  with  references  to  choice  and  self-management  in  W247. 
We  also  recommend  that  HCFA  consider  that  independence  includes  more  exercise  of 
choices,  and  that  active  treatment  should  be  promoting  and  supporting  choice-making. 
This  is  stated  in  guidance  on  when  the  CoP  is  not  met;  we  recommend  that  it  also  be 
stated  explicitly  as  an  indicator  of  compliance. 

Since  the  Comprehensive  Functional  Assessment  (CFA)  and  the  IPP  objectives  are  still 
referenced  as  the  basis  for  assessing  Active  Treatment  compliance  in  the  Revised  Survey 
Procedures  for  Full  Surveys,  surveyors  may  continue  to  consider  the  CFA  and  IPP 
objectives  in  assessing  Active  Treatment  even  if  not  specified.  We  would  therefore 
recommend  that  choice,  independence  and  growth  in  decision-making  as  part  of  the 
CFA  and  objectives  that  underlie  Active  Treatment  provision  and  outcomes  be  explicitly 
stated.  I.e.,  if  choice  and  self-management  outcomes  are  present,  they  begin  in  looking 
at  individual  choices  in  developing  the  CFA  and  IPP.  This  is  also  consistent  with  the 
focus  on  self-determination  and  independence  in  the  IPP  standard  (483.440(c)(4)). 
Individual  participation  as  referenced  in  W209  and  related  guidance  also  may  be  worth 
mentioning  as  a  significant  indicator  as  well. 

The  Certification  Process 

Overview 

On  "Emphasis,  Components  and  Applicability"/1st  paragraph,  would  revise  3rd  sentence 
as  follows: 

The  focus  of  the  survey  is  to  determine  whether  the  facility  is  actually  providing 
active  treatment  and  other  required  services  on  an  individualized  basis  rather  than 
whether  the  facility  is  capable  of  providing  them. 
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I.  INTRODUCTION 


Would  revise  1st  sentence  as  follows: 

This  revised  ICF/MR  survey  protocol  is  to  assist  surveyors  to  focus  attention  on  the 
outcomes  of  Individualized  active  treatment  services. 

4th  paragraph  -  would  add  sentence  on  what  DD  field  is  emphasizing: 

In  addition,  services  in  virtually  all  states  are  placing  increased  emphasis  on 
person-centered  planning  and  person-centered  senyices,  focusing  on  the 
preferences,  goals  and  aspirations  of  each  individual  and  on  supporting  them  in 
reaching  their  personal  goals. 

Also  would  add  re:  QA  trends: 

QA  is  placing  increased  emphasis  on  outcomes  related  to  choice,  control, 
relationships,  community  inclusion  and  satisfaction  with  life,  as  well  a  satisfaction 
with  services  and  supports.  Many  QA  systems  also  include  self-assessment  and 
continuous  quality  improvement  components. 

II.  BACKGROUND 

In  reviewing  CFA  as  the  basis  for  the  IPP,  it  is  appropriate  to  look  ahead  to  self- 
determination  and  independence,  and  therefore  to  add  the  concept  of  identifying 
individual  preferences  and  choices.  These  concepts  are  at  least  referenced  in  current 
W219  guidance  (re:  affective  development)  and  implied  in  W227  et  al.  with  references 
to  individual  objectives.  The  point  is  to  ensure  that  choices  and  preferences  are 
assessed  consistent  with  the  concept  of  individualization. 

These  same  recommendations  also  are  appropriate  for  the  review  of  the  IPP. 

III.  PRINCIPAL  FOCUS  OF  SURVEYS 

The  1st  paragraph  on  page  6  is  on  target  by  talking  about  how  the  individuals,  families 
and  guardians  participate  in  identifying  and  selecting  services.  If  this  is  truly  the  focus 
for  principle  attention,  however,  it  should  be  better  reflected  in  the  discussion  of  the 
standards  for  CFA,  the  IPP,  and  Active  Treatment. 

On  page  7,  the  description  of  methods  is  generally  a  strong  improvement.  We  would 
suggest,  however,  that  at  least  a  cursory/background  review  of  data  on  the  individuals 
in  the  sample  be  made,  for  guidance  in  what  and  how  to  interview,  and  to  look  for  in 
observations.  Would  make  similar  slight  modification  on  page  9,  and  related  clarification 
to  page  19.  I.e.,  making  a  brief  -  not  detailed  -  review  specific  to  identifying  implications 
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for  interviews  and  observations.  This  is  in  addition  to  asking  the  consumer  and  the  direct 
care  staff,  methods  which  we  strongly  support. 

IV.  SURVEY  PROCESS 

The  list  of  CoPs/fundamental  requirements  generally  seems  very  appropriate:  core 
standards  that  also  lend  themselves  to  interview  and  observation.  Recommended 
additions: 

-  W136:  Participation  opportunities  in  social,  religious  and  community  activities. 

-  W1 52:  Re:  criminal  convictions  of  staff. 

-  W240:  Including  promotion  of  independence  in  IPP. 

-  W261 :  Designation  of  committee  to  review  behavior  management  plans. 

-  W264  and  W265:  Responsibilities  of  the  specially  designated  committee. 

-  W286  and  W287:  Prohibition  on  using  medications/restraints  for  discipline  or 
convenience. 

-W316and317:  Re:  gradual  withdrawal  of  psychotropic  medications. 

-  W375  and  376:  Re:  reporting  of  medication  errors. 

In  addition,  we  recommend  a  clarification  that  W436  includes  positioning.  If  this  is  not 
appropriate,  then  W246  is  needed  (consistent  with  retention  of  W153). 

The  revised  approach  described  on  page  9  re:  no  further  review  of  supporting 
requirements  is  very  commendable.  We  would  suggest  clarification,  however,  that 
interviews  and  observations  include  some  on  the  processes  that  are  linked  to  desired 
outcomes,  especially  around  choice,  individual  communication,  preference,  and 
consistency  across  program  areas.  Questions  on  how  choices  are  made,  how 
preferences  are  communicated,  how  individuals  are  learning  how  to  exercise  their  rights, 
etc.,  help  surveyors  get  a  picture  of  much  active  treatment  is,  in  fact,  being  individualized 
and  is  helping  people  become  more  independent  and  have  more  self-direction. 

Re:  the  ICF/MR  recertification  survey  priorities  on  page  9,  it  is  not  clear  how  these  relate 
to  the  previous  paragraph  on  no  further  review  of  supporting  requirements  or  related 
CoPs. 

Re:  triggering  an  Extended  Survey  Process  (p.  9),  we  would  suggest  an  interim  step  of 
conducting  further  investigation  when  the  surveyors  are  not  sure  about  compliance, 
rather  than  going  directly  to  the  extended  process.  Through  using  the  matrix  described 
on  page  40,  and  the  guidelines  on  compliance  in  the  Facility  Compliance  Principles, 
most  decisions  should  be  resolved  without  going  over  all  the  relevant  tags.  Further 
investigation  (additional  interviews  and  observations,  limited  document  review)  should 
be  able  to  resolve  the  rest.  The  concern  would  be  that,  as  presently  worded,  surveyors 
will  revert  to  the  Extended  Survey  when  they  are  short  of  a  clear-cut  decision,  rather  than 
trying  some  further  investigation.  We  would  recommend  as  well  that  the  Extended 
Survey  Process  be  triggered  primarily  when  surveyors  are  doubtful  that  thev  can  find  the 
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ICF/MR  in  compliance,  using  the  full  set  of  relevant  tags.  This  is  responsive  to  surveyors' 
concerns  that  they  may  need  to  have  extensive  documentation  when  citing  non- 
compliance on  a  CoP,  without  encouraging  full-blown  traditional  surveys  (especially  on 
Active  Treatment)  just  because  of  a  few  unresolved  questions.  This  recommendation 
also  is  made  for  the  description  of  Recertification  Surveys  on  page  11. 

The  guidance  ("NOTE")  at  the  bottom  of  page  10  is  important.  We  recommend  that  it  be 
moved  up  to  somewhere  in  the  opening  discussion  of  the  different  types  of  surveys.  It 
could  be  combined  with  the  recommendation  in  the  previous  paragraph  on  conducting 
further  investigation  whenever  there  are  questions  on  the  core  areas  of  compliance. 

VIII.  TASK  3:  DIRECT  INDIVIDUAL  OBSERVATIONS 

The  focus  on  observations,  including  off-site  program  areas,  is  very  appropriate.  In  line 
with  previous  recommendations,  we  suggest  clarification  that  a  cursory  record  review  of 
individuals  in  the  sample  may  be  helpful,  just  not  a  detailed  one. 

Re:  determining  if  the  necessary  relationship  between  individual  needs  and  what  staff 
know  and  do  with  individuals  is  being  made  (page  19),  would  recommend  adding 
individual  needs  and  preferences.  An  example  of  guidance  to  surveyors  on  reviewing 
active  treatment  and  behavioral  programming  that  includes  looking  at  individual 
preferences  from  Module  6D  of  surveyor  training  materials  (HCFA,  1993)  is  attached. 

IX.  TASK  4  -  INTERVIEWS 

Here  (and  elsewhere  where  interviews  are  discussed)  we  recommend  the  addition  of 
some  guidance  on  using  interviews  with  people  who  have  very  limited  or  challenging 
communication.  In  large  public  ICFs/MR,  for  example,  where  80-90  percent  of  the 
individuals  may  have  severe  or  profound  mental  retardation,  interview  procedures  will 
need  to  be  modified,  and  surveyors  (and  the  ICF/MR)  need  to  be  looking  at 
communication  through  multiple  modes.  We  recommend  using  the  ICF/MR  Surveyor 
Training  Module  on  communication  for  suggestions. 

XII.  TASK  7  -  RECORD  REVIEW 
These  are  very  helpful  revisions. 

XIII.  TASK  8  -  TEAM  ASSESSMENT  OF  COMPLIANCE 

On  page  30,  re:  additional  information  and/or  documentation,  we  recommend  that  HCFA 
consider  suggesting  additional  interviews:  family  members,  guardian,  friend-advocates, 
etc.,  as  well  as  possibly  interviewing  additional  consumers  and  direct  care  staff  members. 
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On  page  31,  we  urge  that  the  Facility  Compliance  Principles  (and  possibly  also  the 
Decision-Making  matrix)  apply  to  the  Fundamental  as  well  as  to  the  Extended  and  Full 
Surveys. 

Re:  behavior  management  approaches  on  pp.  32-33,  we  recommend  adding  questions 
on  reliance  on  aversives,  the  absence  of  positive  teaching  approaches,  and  the  lack  of 
opportunities  for  independence,  choice  and  decision-making. 

Re:  environmental  importance  on  page  32,  we  recommend  adding  a  question  on 
individual  environmental  preferences  and  possible  interference  with  active  treatment,  as 
well  as  the  significance  of  assistive  technology,  adaptive  devices/equipment  and 
positioning. 

On  page  34,  the  reference  to  "practice  standards"  is  unclear. 

CONDITION  LEVEL  DECISION  MAKING 

The  matrix  and  definitions  on  pp.  40-41  generally  seem  appropriate.  We  suggest  that 
HCFA  consider  adding  examples  of  how  the  matrix  would  be  used  in  relation  to  different 
CoPs,  as  well  as  clarification  on  what  is  meant  in  the  definition  of  severe  re:  "it  is  likely 
those  negative  outcomes  will  result  in  the  near  future." 

Facility  Compliance  Principles  -  see  above. 

Revised  Survey  Procedures  -  W195  -  W196 

This  generally  seems  appropriate,  however,  it  is  not  clear  how  it  is  intended  to  be  used. 
We  would  recommend  adding  "privacy"  and  "relationships"  under  Rights,  "choices  and 
preferences  determined?"  and  "positive  teaching  strategies?"  under  Program. 

Suggested  Interview  Questions 

As  noted  elsewhere,  we  suggest  that  HCFA  provide  considerable  additional  guidance, 
including  examples,  of  how  to  do  alternative  interview-type  data  collection  with 
consumers  with  severe  and  profound  mental  retardation  and  limited  or  challenging 
communication.  Alternatives  should  include  discussions  with  direct  care  staff,  family 
members  and/or  friend  advocates  or  guardian  about  how  the  facility  determines 
individual  choices  and  preference,  strengths,  personal  goals  and  interests,  etc.,  and  how 
these  are  respected  and  built  upon  in  assessing  and  promoting  individual  rights,  and  in 
delivering  active  treatment.  We  also  recommend  that  a  very  brief  review  of  the 
consumer's  record  be  made  for  some  quick  guidance  on  communication  modes.  As 
noted  above,  we  suggest  that  HCFA  incorporate  some  of  the  suggestions  in  the  ICF/MR 
Surveyor  Training  module  on  communication,  including  the  videotape.  These  materials 
cover  communication  guidelines  for  surveyors  in  interviewing  consumers  as  well  as 
suggestions  on  what  to  look  for  in  staff  interactions  and  descriptions  provided  by  staff 
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re:  communication  and  their  understanding  of  individual  preferences  and  needs 
expressed  by  consumers  with  challenging  communication. 

We  recommend  adding  questions  to  staff,  especially  direct  care  staff,  in  each  area 
covered,  and  to  friend/advocates  when  consumers'  circles  of  support  includes  such 
people.  Some  additional  suggestions  on  specific  sections  are  as  follows: 

1.  W1 24-1 28 
Individual  Served 

We  recommend  expanding  more  on  the  question  of  things  the  person  would  like  to  do 
more  often.  Re:  W125,  we  suggest  asking  more  on  individual  interests  and  choices. 
Language  also  should  be  modified  on  "going  to  church,"  recognizing  that  this  term  is 
limited  to  specific  religious  faiths.  We  also  would  note  that  the  Constitutional  right  to 
freedom  of  religion  is  more  than  just  going  to  services.  The  last  three  questions  on 
money  seem  more  appropriate  to  W-126. 

Family  Member 

We  recommend  adding  a  question  re:  W124  on  who  provides  consent  when  needed, 
the  individual  or  family  member/guardian,  and  if  consent  has  been  needed  and  obtained 
for  any  programs  or  treatments.  We  also  suggest  asking  them  if  there  have  been  any 
incidents  involving  possible  abuse  or  punishment,  and  if  so,  were  they  able  to  obtain 
sufficient  information,  and  was  the  concern  addressed. 

2.  W126 

In  addition,  would  recommend  adding  questions  for  W1 27-1 28  on  abuse/punishment, 
unnecessary  drugs  and  physical  restraints. 

3.  W133,  W146 

In  this  same  area,  we  recommend  adding  W136  to  list  of  tags  in  Fundamental  Survey 
and  adding  questions  here  on  participation  in  social,  religious  and  community  activities. 

4.  W143  [also  seems  to  relate  to  W209] 

We  recommend  adding  the  following  question: 

If  not  attending  the  meeting,  how  were  your  views  on  goals,  preferences,  personal 
needs  and  choices  expressed  and  included  in  the  Interdisciplinary  Team  meeting? 
[modified,  as  needed,  depending  on  level/style  of  communication] 
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For  family  members,  we  recommend  adding  a  question  on  the  team's  receptivity  and 
attention  to  their  relative's  (i.e.,  the  consumer's)  views,  how  their  choices  were  used, 
extent  to  which  the  individual's  goals  and  preferences  were  the  focus  of  the  team 
meeting's  discussion,  etc. 

5.  W144 
Individual  Served 

The  question  seems  to  fit  more  with  W127. 

6.  W147  and  8.  W247. 

These  questions  also  seem  relevant  to  W136,  which  we  recommend  adding  to  the 
Fundamental  Survey,  re:  participation  in  community  and  social  activities  and  having 
relationships. 

9.  W263 

Individual  Served 

This  question  seems  too  parental  in  tone.  It  is  also  not  clear  how  this  relates  to  W263. 
Family  Member 

We  recommend  adding  questions  on  any  use  of  restraints  or  other  aversive  stimuli,  and 
possibly  asking  an  open-ended  question  on  what  kinds  of  positive  teaching  strategies 
(with  example)  are  being  tried. 

Overall,  this  section  seems  to  need  much  more  work  than  the  other  components  of  the 
revised  survey  procedures. 
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EXCERPT  FROM  MODULE  6D 
ICF/MR  INSTRUCTOR'S  MANUAL 
FALL  1993 
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Behavior  doesn't  just  "happen."  Internal  or  external  factors  -  or  a  combination  of  the  two 
-  trigger  a  set  of  responses  to  a  given  event  or  stimulus.  By  changing  the  context, 
changing  the  stimulus  or  its  presentation,  or  modifying  what  happens  after  the  behavior 
occurs,  the  behavior  itself  can  be  changed. 

Let's  consider  a  typical  scenario.  Every  evening  you  watch  television  at  7  p.m.  for  half 
an  hour.  During  this  time  you  see  at  least  three  commercials  for  food.  Tonight  you  see 
a  "bake  someone  happy"  commercial  for  a  chocolate  cake.  After  seeing  the  commercial 
you  decide  that  you  want  something  to  eat  and  go  into  the  kitchen  in  search  of 
something  resembling  chocolate  cake.  To  eat  the  cake,  you  will  have  gone  through  at 
least  some  of  the  following  steps: 

1.  Decided  that  you  wanted  something  to  eat 

2.  Decided  that  you  wanted  to  eat  more  than  you  wanted  to  continue  watching  television 

3.  Moved  from  where  you  were  watching  television  to  the  kitchen 

4.  Searched  for  food 

5.  Determined  what  you  wanted  to  eat 

6.  Prepared  it,  if  necessary 

7.  Ate  the  food 

Behavior  is  prompted  by  a  chain  of  events  or  activities.  One  way  to  examine  behavior, 
particularly  inappropriate  behaviors  exhibited  by  people  with  mental  retardation  or  other 
developmental  disabilities,  is  the  ABC  chain,  or  Antecedent,  Behavior  and  Consequence 
chain. 

How  many  of  you  are  familiar  with  the  ABC  chain? 

Option:  ask  participants  who  indicate  that  they  are  familiar  with  the  ABC  chain  to 
describe  it  briefly  -  being  prepared  for  the  possibility  that  their  information  may  be 
somewhat  different  from  what  is  on  the  handout. 

Distribute  Handout  6D-1  and  show  Lecture  Guide  6D-1.  Use  the  Lecture  Guide  to 
highlight  the  following  points  on  the  ABC  chain. 

Antecedent  refers  to  a  stimulus  which  precedes  a  behavior.  The  stimulus  is  consistently 
present  before  the  behavior  occurs  and  is  likely  to  influence  the  behavior.  In  the 
chocolate  cake  example,  the  television  commercial  was  the  antecedent.  It  set  up  the 
chain  of  events  that  followed.  Antecedents  may  include  noise,  fatigue,  smells,  or  events. 
For  example,  a  crowded  room  filled  with  people  making  loud  noises  may  be  the 
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antecedent  for  a  client  to  withdraw  and  engage  in  self-stimulating  behaviors  or  become 
agitated  and  disruptive.  The  stimulus  may  be  internal  or  external,  may  be  volitional  or 
influenced  by  reflexes,  chemical  imbalances,  medication,  allergies,  or  seizures. 

Behavior  is  the  observable  response  an  individual  makes  to  a  stimulus.  In  the  example 
of  the  television  viewer,  the  behavior  was  to  stop  watching  television,  search,  prepare, 
and  eat  food. 

Consequences  are  stimuli  that  occur  contingently  following  a  particular  response.  These 
payoffs,  incentives,  or  motivators  can  be  manipulated  to  encourage  or  discourage 
behaviors  or  skills.  As  we  discussed  in  the  module  on  the  basics  of  changing  behavior, 
the  payoffs  for  the  desired  behavior  must  have  more  appeal  or  power  than  the  payoffs 
for  the  undesirable  behavior,  in  order  for  the  latter  behavior  to  be  changed. 

For  people  with  mental  retardation  or  other  developmental  disabilities,  acting  out 
behaviors  may  be  their  learned  way  of  expressing  pain,  discomfort,  anger,  etc. 
Therefore,  internal  factors  such  as  pain,  frustration  or  an  allergic  reaction  must  be 
considered  as  well  as  external  factors  (such  as  staff  or  other  client  interactions,  strong 
odors  or  allergens  in  the  room  where  the  behavior  took  place)  before  a  program  to 
decrease  targeted  behaviors  can  be  implemented. 

An  important  point  to  remember  is  that  our  behavior,  and  that  of  other  individuals,  serves 
a  purpose.  Suppose  the  television  viewer  is  on  a  diet  -  let's  pretend  that  it's  a  woman  - 
and  eating  between  meals  is  an  undesirable  behavior.  The  behavior  may  be  pointing 
to  a  larger  problem  of  low  self-esteem,  or  she  may  be  using  food  to  stuff  feelings  that  are 
too  painful  to  express.  We  know  that  this  pattern  does  not  represent  normal  eating, 
because  at  no  point  in  the  sequence  of  events  did  she  stop  and  ask  herself,  "Am  I 
hungry?"  In  fact,  asking  this  question  could  have  been  the  intervention  needed  to 
prevent  overeating. 

To  change  behavior,  it  is  necessary  to  identify  the  antecedents,  observe  the  behavior, 
and  determine  consequences  that  will  result  in  a  desirable  change  in  behavior  rather 
than  maintenance  of  the  behavior.  For  example,  our  dieter  may  not  lose  the  desired 
weight  if  she  eats  the  chocolate  cake.  However,  if  being  overweight  reinforces  the  low 
self-esteem  of  the  dieter,  a  conflict  of  payoffs  is  present.  If  low  self-esteem  is  long 
standing,  feeling  good  about  oneself  may  not  be  enough  motivation.  To  change 
undesirable  behaviors  it  is  necessary  to  determine  interventions  that  will  work,  that  is, 
interventions  that  will  result  in  the  desired  behavior.  Persistence  is  required  to  determine 
the  consequences  which  will  alter  the  person's  behavior. 
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Handout  6D-1 


DEFINITIONS 


Antecedent:  A  stimulus  which  precedes  a  behavior.  The  stimulus  is 

consistently  present  before  the  behavior  occurs  and  is  likely 
to  influence  the  behavior. 


Behavior:  The  observable  response  an  individual  makes  to  a  stimulus. 


Consequences:  Stimuli  that  are  presented  contingently  following  a  particular 
response. 
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Distribute  Handout  6D-2  and  show  Lecture  Guide  6D-2.  Use  the  Lecture  Guide  to 
highlight  key  points  on  the  components  of  assessment. 

Another  approach  to  looking  at  behavior  is  functional  analysis.  Functional  analysis 
recognizes  that  behavioral  programs  may  need  to  look  more  broadly  than  the  immediate 
apparent  antecedent  to  the  behavior.  For  example,  although  the  TV  commercial  on 
chocolate  cake  was  the  immediate  antecedent  to  the  dieter's  getting  a  snack,  the 
behavior  also  was  associated  with  how  she  was  feeling  that  evening,  her  attempts  at 
dieting  (current  and  past),  and  her  sense  of  self-esteem.  Assessments  in  behavioral 
programs  for  people  with  mental  retardation  and  other  developmental  disabilities  need 
to  consider  all  aspects  of  the  individual's  life  in  order  to  determine  the  most  effective 
supports  and  interventions,  including: 

Scheduling  and  Structure 

How  well  does  the  schedule  and  sequence  of  events  match  the  person's 
individual  "rhythm?"  (We  are  not  all  morning  people.) 

How  structured  and  demanding  are  the  situations?  (Some  of  us  do  better 
with  lots  of  structure  than  others.) 

Opportunities  for  Choice  and  Control 

Does  the  person  have  opportunities  to  make  any  choices?  To  have  some 
control  over  the  environment?  (Most  of  us  don't  like  feeling  completely 
helpless  or  hemmed  in.) 

Communication 

How  does  the  person  let  people  know  wants  and  choices? 

How  can  we  observe  carefully  what  people  are  feeling?  (Most  people  get 
really  frustrated  when  they  need  something  or  are  feeling  strongly  about 
something  and  no  one  seems  to  understand.) 

Relationships 

How  does  the  person  respond  to  the  various  people  in  his  or  her  life? 

What  kinds  of  interaction  seem  to  help  the  person  react  more  positively? 
(Most  of  us  work  better  or  enjoy  being  with  some  people  more  than 
others.) 
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Handout  6D-2 
FUNCTIONAL  ANALYSIS  COMPONENTS 


Scheduling  and  Structure 

How  well  does  the  schedule  and  sequence  of  events  match  the  person's 
individual  "rhythm?"  (We  are  not  all  morning  people.) 

How  structured  and  demanding  are  the  situations?  (Some  of  us  do  better 
with  lots  of  structure  than  others.) 

Opportunities  for  Choice  and  Control 

Does  the  person  have  opportunities  to  make  any  choices?  To  have  some 
control  over  the  environment?  (Most  of  us  don't  like  feeling  completely 
helpless  or  hemmed  in.) 

Communication 

How  does  the  person  let  people  know  wants  and  choices? 

How  can  we  observe  carefully  what  people  are  feeling?  (Most  people  get 
really  frustrated  when  they  need  something  or  are  feeling  strongly  about 
something  and  no  one  seems  to  understand.) 

Relationships 

How  does  the  person  respond  to  the  various  people  in  his  or  her  life? 

What  kinds  of  interaction  seem  to  help  the  person  react  more  positively? 
(Most  of  us  work  better  or  enjoy  being  with  some  people  more  than 
others.) 
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The  key  to  assessment  is  having  a  full  picture  of  what  is  affecting  the  behavior  of  a 
particular  individual.  From  that  assessment,  an  individualized  behavioral  program  plan 
can  be  designed  that  looks  at  all  the  ways  to  decrease  the  factors  associated  with  the 
undesirable  behavior  and  increase  all  the  factors  associated  with  the  desired  behavior. 

The  objective  in  decelerating  (reducing)  targeted  behaviors  is  not  solely  the  reduction  of 
inappropriate  behaviors.  The  objective  is  the  deceleration  of  inappropriate  behaviors  and 
their  replacement  with  positive  and/or  adaptive  behaviors.  Have  you  ever  noticed  how 
when  you  first  try  to  eliminate  negative  thoughts  from  your  mind,  the  number  of  negative 
thoughts  you  think  actually  increases?  That's  because  you  didn't  replace  them  with 
positive  thoughts.  Not  replacing  negative  thoughts  with  positive  thoughts  is  a  reason 
positive  mental  attitude  programs  don't  work  for  many  people.  They  only  do  half  the 
required  task,  and  the  vacuum  created  is  filled  with  more  negative  thoughts,  rather  than 
positive  ones. 

The  same  is  true  for  behaviors.  If  an  inappropriate  behavior  is  not  replaced  with  adaptive 
behavior,  there  is  likely  to  be  an  increase  in  inappropriate  behavior(s).  For  example, 
consider  an  individual  who  engages  in  compulsive  talking  with  non-productive  speech. 
A  program  which  reinforces  appropriate  verbalization  while  reducing  inappropriate 
verbalization  should  be  implemented.  This  program  could  include  behavior  modification 
techniques  which  reduce  the  inappropriate  verbalizations  as  well  as  to  increase  social 
interactions  to  provide  opportunities  to  verbalize  appropriately. 

Behaviors  do  not  exist  in  a  vacuum.  There  are  factors  which  precipitate  and  contribute 
to  their  occurrence.  Uncovering  the  antecedents  of  behavior  requires  investigative  work. 
Successful  redirection  of  behavior  depends  on  identification  of  the  antecedents  and  the 
creation  of  strategies  to  effectively  deal  with  the  stimuli.  If  people  are  not  making 
progress,  are  getting  worse,  or  develop  new  bizarre  behaviors  in  response  to  behavior 
modification  strategies,  the  strategies  must  be  re-evaluated.  This  is  true  even  if  the 
strategy  includes  nice,  positive,  socially  acceptable  types  of  reinforcers.  If  you  examine 
an  IPP  and  find  that  clients  are  not  progressing,  something  is  wrong  with  the  program 
and  further  investigation  is  needed  to  determine  what  changes,  if  any,  are  being  made 
and  why  they  are  believed  to  have  change  potential. 

Let's  look  at  an  example  of  how  inappropriate  behaviors  can  be  eliminated  and  replaced 
with  adaptive  behaviors. 

Suppose  a  client  engages  in  severe  head-banging.  A  new  psychologist  comes  to  the 
facility  and  decides  to  implement  a  behavioral  program  to  change  the  behavior.  The  first 
strategy  implemented  provided  reinforcement  of  "no-head  banging"  with  candy.  The 
candy  was  offered  intermittently  during  periods  of  non-head  banging.  (This  type  of 
reinforcement  is  called  intermittent  scheduled  reinforcement.)  If  the  client  didn't  bang  his 
head  for  a  period  on  the  average  of  1 5  minutes,  he'd  receive  a  piece  of  candy. 
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Over  a  couple  of  days  the  client  began  to  make  an  association  between  head  banging 
and  candy.  However,  he  increased  the  head  banging,  apparently  in  order  to  receive  the 
candy.  Obviously,  this  strategy  wasn't  working.  The  wrong  association  was  made  -  that 
candy  was  a  reward  for  head  banging. 

The  next  strategy  tried  involved  hugging  and  praising  the  client  when  he  stopped  head 
banging.  This,  too,  increased  head  banging.  The  client  apparently  thought  that  if  he 
increased  his  head  banging  he  would  receive  more  hugs  and  praise.  Other  strategies 
were  implemented  until  a  successful  one  was  found  which  incorporated  reinforcement 
for  not  head  banging,  diversion  to  hand  clapping  whenever  head  banging  began,  and 
a  planned  program  of  tactile  stimulation  immediately  prior  to  the  times  head  banging  had 
been  found  to  occur.  This  example  shows  that  the  caregivers  must  be  persistent  and 
a  variety  of  techniques  must  be  explored  to  change  behaviors.  The  facility  could  have 
just  given  up  and  said  that  this  client  was  never  going  to  stop  head  banging.  Instead 
they  persevered  and  found  an  intervention  which  was  successful  in  reducing  this 
potentially  life-threatening  behavior. 

Some  behaviors  of  clients  are  inappropriate  only  with  regard  to  where  the  behavior 
occurs.  For  example,  clients  masturbating  privately  is  considered  far  more  adaptive  and 
appropriate  than  masturbation  conducted  in  public.  Clients  can  be  taught  where  and 
when  behaviors  are  appropriate.  Again,  behavioral  programs  based  on  assessment  can 
be  used  to  accomplish  these  goals. 

The  video  we're  about  to  see  looks  at  David,  at  a  point  in  his  life  when  he  needed  a  great 
deal  of  help  in  dealing  with  his  behavior. 

Begin  Video  6D,  Real  Teaching  -  Part  II. 
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April  30, 1996 


Anthony  J.  Tirone 
Deputy  Director 

Health  Standards  and  Quality  Bureau 
Health  Care  Financing  Administration 
S2-11-07  South  Building 
7500  Security  Boulevard 
Baltimore,  MD  21244-1850 

Dear  Mr.  Tirone: 

Thank  you  for  providing  us  with  the  opportunity  to  comment  on  the  revised 
Conditions  of  Participation  (CoPs)  for  intermediate  care  facilities  serving  people  with 
mental  retardation  and  related  conditions  (ICFs/MR).  We  congratulate  you  and  your 
staff  for  developing  a  revised  approach  that  is  clearly  more  in  keeping  with 
contemporary  notions  of  individualized  supports  and  an  attention  to  the  outcomes  of 
service  interventions. 

As  you  know,  we  have  been  working  with  your  staff  on  a  project  exploring  innovative 
quality  assurance  approaches  around  the  country.  You  are  currently  in  receipt  of  our 
draft  report.  In  that  report,  we  discuss  possible  directions  for  HCFA  that  would  move 
even  further  toward  an  outcome-focused  monitoring  approach  and  conception  of  active 
treatment.  With  this  in  mind,  we  offer  the  following  comments  which  should  also  be 
viewed  in  the  larger  context  of  our  report. 

Again,  we  feel  that  the  changes  proposed  are  positive  and  propose  the  following 
suggestions  as  constructive  recommendations  based  on  our  broad  contact  with  a  range 
of  people  around  the  country  concerned  with  the  quality  of  care  and  services  for  people 
with  mental  retardation. 

Sincerely  yours, 

Valerie  J  Bradley 
President 
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Prepared  by  Human  Services  Research  Institute 
and  Jaskulski  and  Associates 

COMMENTS  ON  REVISED  CONDITIONS  OF 
PARTICIPATION  FOR  ICFS/MR 


CoP  #1:      Individual  Rights  Are  Promoted  and  Protected, 
(a)  Rights  Outcomes  for  Individuals 

Recommend  adding  a  standard  on  providing  informed  consent,  especially  to  any 
behavioral  programs  with  negative  consequences.  Although  this  is  covered  in  CoP  #3 
on  interventions  to  manage  targeted  behaviors,  providing  consents  is  a  basic  right  that 
includes  health  care  and  program  interventions  beyond  just  those  focused  on  behavior 
management. 


(b)  Individual  Rights  Facility  Processes 


Recommend  adding  the  following: 

•  The  facility  implements  a  system  for  obtaining  informed  consent,  including 
supports  to  increased  understanding  of  the  individual  in  providing  informed 
consent 

•  The  facility  obtains  or  supports  an  external  system  of  obtaining  legal  and  friend 
advocates  for  individuals  who  need  or  request  such  advocacy. 

•  The  facility  implements  a  system  to  educate  individuals  regarding  their  rights 
as  will  as  supporting  them  in  exercising  their  rights. 

•  The  facility  supports  individuals  to  exercise  their  freedom  of  choice  rights  in 
selecting  service  providers  and  alternate  living  arrangements. 
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CoP  #2:      Individuals  Receive  the  Supports  and  Services  Needed  to 
Increase  Functional  Skills  and  Independence. 

Recommend  rewording  to  read:  Individuals  Receive  the  Supports  and  Services  Needed 
to  Meet  Personally-Defined  Goals  (or  equivalent) 

(a)  Active  treatment  definition  -  Recommend  that  this  section  be  re-titled: 
Individualized  Services  and  Supports  (Active  Treatment)  Definition  and  rewritten  as 
follows:. 

(1)  Each  individual  must  receive  individualized  services  and  supports  (active 
treatment)  based  on  personal  goals,  preferences  and  needs.  The  individualized 
active  treatment  program  must  be  implemented  consistently  and  include 
services  and  supports  needed  by  each  individual  to: 

•  Develop  increasing  skills,  especially  those  that  promote  greater 
independence,  self-determination,  and  achievement  of  goals  important  to  the 
individual; 

•  Develop  and  maintain  satisfying  relationships; 

•  Participate  in  community  life; 

•  Exercise  rights  and  responsibilities; 

•  Engage  in  productive  activities;  and 

•  Enjoy  and  participate  in  social,  leisure  and  recreational  activities. 

(2)  Active  treatment/ individualized  services  and  supports  must  prevent  or 
decelerate  regression  or  loss  of  current  optimal  functional  status. 

(3)  [same  as  current  #2] 

(b)  Active  Treatment  Outcomes  for  Individuals 

Recommend  putting  as  the  first  outcome  the  following: 
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"Each  individual  receives  individualized  services  and  supports  (active  treatment 
program)  that  is  based  on  their  person-centered  plan,  in  which  the  individual 
and  those  who  know  him/her  best  have  identified  personal  goals  and  the 
services  and  supports  need  to  reach  those  goals." 

Move  #6  to  follow  this  outcome. 

(c)  Active  Treatment  Facility  Processes 

Revise  #4  as  indicated  and  put  first: 

'The  facility  implements  a  system  to  ensure  that  individual  personal  goals  and 
desires  are  identified  and  form  the  basis  of  his/her  plan  of  services  and 
supports." 

Add  at  end  of  #7: 

"and  to  reduce  the  need  for  behavior  management  programs." 
Add  new  process: 

"Supports  are  provided  to  help  individuals  develop  and  maintain  relationships, 
to  increase  their  participation  in  community  life,  and  to  have  experiences  that 
increase  their  ability  to  make  choices." 

Revise  #10  as  follows: 

The  facility  ensures  that  individual  plans  are  modified  to  changed  in  response  to 
the  individuals  specific  accomplishments,  changes  in  individual  goals  and 
preference,  need  for  new  programs,  or  difficulties  in  acquiring  or  maintaining 
skills. 

CoP  #3  on  Behavioral  Interventions 
(a)  Outcomes  for  individuals 

Recommend  adding: 


D-20 


"Each  individual  receives  services  and  supports,  including  modifications  of  the 
environment,  to  reduce  or  eliminate  the  need  for  such  interventions." 

(b)  Facility  process  standards 

Recommend  adding  reference  to  the  specially  constituted  committee  as  in  current 
standards,  and  the  following: 

'The  facility  makes  information  on  policies  and  any  implementation  of  behavior 
management  programs  readily  available  to  advocates  and  legal  guardians  of 
affected  individuals  and  the  state's  designated  Protection  and  Advocacy 
agency." 

Cop  #4  Health 

(a)  Outcomes  for  individuals 

Recommend  adding: 

"Each  individual  participates  in  his/her  own  health  care  to  the  maximum  extent 
possible  and  receives  support  in  taking  increased  responsibility  for  maintaining 
the  best  possible  health." 

(b)  Facility  process  standards 

Recommend  adding: 

"The  facility  provides  health  education  and  other  supports  to  individuals  to 
assist  them  in  developing  and  maintaining  the  best  possible  health  and  to 
increase  their  ability  to  take  responsibility  for  their  health  care." 

Additional  Comments 

We  feel  that  Safety  may  not  have  enough  visibility.  It  possibly  could  be  combined  with 
health,  or  added  to  Rights  or  addressed  more  clearly  in  CoP  #5. 

Recommend  Individual  Outcome; 
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"Each  individual  receives  services  in  a  safe  environment  and  supported  in  living 
safely." 


Recommend  Facility  Process: 

'The  facility  determines  the  safety  needs  for  each  individual,  and  provides 
education  and  supports  on  an  individualized  basis  to  promote  each  person's 
safety  and  their  ability  to  take  increasing  responsibility  for  their  personal 
safety." 
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APPENDIX  E 
CUMULATIVE  LIST  OF  ALL  INTERVIEWEES 


CUMULATIVE  LIST  OF  ALL  INTERVIEWEES 
(Asterisk  *  indicates  the  individual  offered  comments  about  ICF/MR) 


Developmental  Disabilities  Services: 


Association  for  Community  Living: 


COPAR  interviewer: 

Colorado  Department  of  Public 
Health  and  Environment: 


Legal  Center: 

People  First  of  Colorado: 


Colorado  Association  of  Community- 
Centered  Boards: 


Jefferson  County  CCB: 

Johnstone  Developmental  Center: 
Foothills  Developmental  Center 


Good  Shepherd  ICF/MR: 
Developmental  Disabilities  Council: 


Colorado 


Kerry  Stern* 
Judy  Ruth 
Lynne  Struxness 
Heidi  Cunningham 

Molly  Markert* 
Tim  Cairns* 
Linda  Skaflen* 

Dottie  Balka* 


Sharon  Haney* 
Terry  Zamell 
Marilyn  Waldschmidt 

Mary  Anne  Harvey* 

Anastasia  Lawhead 
Jill  Karet 
Douglas 
Nick 

Self-advocates  from  Boulder 


Art  Hogling* 
Chris  Collins* 

Jane  Byron* 
Judy  Pauley* 
Casey  Glaser* 

Julie  Bielensen* 

Deborah  Lapp 
Erin  Strack 
Phyllis  Freeman 
Connie 

Cindy  Warren* 
Barbara* 

Don  St.  Lewis* 

Oklahoma 
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Develpmental  Disabilities: 
McClain* 
Services  Division 
Goddell* 

Moore* 


Governor's  Performance  Team: 
Oklahoma  City  providers: 

Department  of  Health: 
Oklahoma  AIM: 
monitors 
Tulsa  providers: 
Oklahoma  AIM: 

monitors 

DDSD  Area  2: 
Hitchye 

DHR  Office  of  Client  Advocacy: 

Oklahoma  State  University: 
Dodder 

Fullerton 


Office  of  Developmental: 
Disabilities  Services 


Jim  Nicholson*  Gwen 

Dennis  Bean*  David 

Bob  Nort*  Priscilla 

Alice  Gord*  Leslie  Parks* 

Makela  Bishop*  Sandy 

Leslie  Bonita 
Deborah 

Greg  Thomas 

Judy  Goodwin*,  OCP 

Terry  Vaughan*,  Center  of  Family  Love 

Belinda  Stevens*,  Edmonds  ARC 

Linda  Wilkerson* 

Brenda  Zyzack  Sandra,  staff 

3  consumer  monitors  4  parent 


Frank  Hackett,  VOA  Kay,  VOA 

Deanne  Aimsley  Chris 

Mary  Ann  Duncan  John  Gajda 

1  consumer  monitor  5  parent 


Gay  McMillen  Sammye 
Tami  Kelley  Harry 
Chase  Gordon 

Barbara  Murray  Richard 

Pat  Bell  Amanda 

3  surveyors 

Oregon 

James  Toews*.  director 
Mary  Lee  Fay*,  QA  manager 
Sean  McMullen 
Sue  Stoner 
Julie  Beaton 
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ODDS  Licensing  staff: 


Holly  Robinson 
Terry  Ritner 
Sue  Birge 
Barb  Southard 


Roland  Brown 
Jim  Ransom 
Gretchen  Thompson 


Advocates: 


Oregon  Technical 
Assistance  Corporation: 

Fairview  Training  Center: 


Charlotte  Duncan*,  DD  Council 

Katherine  Weit*.  DD  Council 

K.  Meeks*,  Arc  of  Oregon 

Janna  Starr*,  Oregon  Advocacy  Center 

Mike  Maley* 
Ron  Spoelstra* 

Chuck  Farnham* 
Laurie  Lindberg* 
Audra  Hamilton* 
Jimmy  Clark* 


State  Training  Center  Review  Board 
AIM  Monitors: 


Penny  Spaccoratelli1 

Phyllis  Crocker 
Marilyn  Bell 
Betty ... 

Marty  McKean,  staff 


Evelyn  Henderson 
Teli  Ragsdale 
Candy ... 

Julie  Cumiford,  staff 


Providers: 


Paradigm  Systems: 
University  of  Oregon 


Zellee  Allen 
Jan  Krai 
Vicki  Harbaugh 
Dan  Guevara 
Bill  Adams 

Darla  Wilson 

Rob  Horner 
David  Mank 


Ken  Becker 
Anson  Bell 
Rick  Curren 
Roger  Hassenpflug 


Billy  Ard 


Oregon  Rehabilitation  Association:  Tim  Krai 
Marion  County  DD  agency:  Mike  Sedlock, 


Senior  and  Disabled 
Services  Division: 


Shirley  Saries* 
Ruth  Tooze* 


Alice  Wagner* 


Utah  Site  Visit  Interviewees 

Division  of  Services  to  People        Georgia  Baddley*  Sharon  Yearsley 

with  Disabilities:  Sue  Geary*  Sue  Marquardt* 

Renae  Taylor*  Robyn  Mendenhall* 

Nonie  Lancaster  Sylvia  Wright 
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Provider,  designed  survey: 

Dave  Hennessey 

DHS  Division  of  Licensing: 

Alan  Hayward 

Utah  Health  Department: 

Ann  E.  Lee* 

Deb  WynkoopGreen 

Allan  Jenkins* 

Sara  Sinclair* 
Helene  Middleton 

Salt  Lake  City  providers: 

Joyce  Hailing* 
Marsha  Parisi 
Phil  Shumway 

Ann  Walker 
John  Harbert 

Advocates: 

Cathy  Chambless*,  GCPD 
Eric  Mitchell*,  Legal  Center 
Bev  Adcock*,  ARC 
Deborah  O'Dell*.  DSPD 

Utah  State  Developmental  Center: 

Keith  Stroud* 
Jim  Jex* 

Karen  Clarke* 
Vicki  Gurney* 

USDC  volunteer  monitor: 

LaReal  Mace* 

Mental  Retardation  Association 
of  Utah  (parent  advocates): 

Dean  Robinson* 
Carola  Zitzmann* 

Claudia  Miles* 

Provo  providers: 

Cathy  Richins* 
Bonnie  Duval* 
Richard  Pace 

Kenneth  Ekong* 
Gerald  Nebeker* 
Mary  Pace 

Western  Region  DSPD  staff: 

Eve  Hendrix 
Terri  Jensen 

Bob  Brewer 
Ana  Manent 

Western  Region  vol.  monitor: 

Mary  Gritton 

Western  Region  families: 

Joyce  Parker 
Judy  Moody 

Carol  Allen 

Legislators: 

Brent  Haymond* 
Judy  Ann  Beth  may  er* 

Charles  Stewart* 

Central  Region  DSPD  staff: 

Linda  Worsley 
Ron  Niederhauser 

Ken  Tidwell 

Central  Region  vol.  monitor: 

Marilyn  Williams 

Central  Region  families: 

Bobbie  Saunders 

Mary  Paulsen 

Legislative  Fiscal  Office: 

J.  Winslow 

North  Dakota  (telephone  only) 

North  Dakota  (Grafton)  ICF/MR: 

Joanne  Steffen 
Wyoming 

Division  of  Developmental 
Disabilities: 

Jon  Fortune* 
Robert  Clabby* 

Nancy  Riley* 
Wayne  Johnson* 
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University  Affiliated  Program: 

Health  Quality  Planning  and 
Program  Evaluation: 

DD  Council: 

Magic  City  Enterprises: 

The  Ark  (Laramie): 


Protection  &  Advocacy: 

Arc  of  Cheyenne,  Arc  of  Casper: 

Legislator: 

Community  Entry  Services: 


Mountain  Regional  Services: 
Weston  Class  Advocates: 

Legislator: 

Wyoming  State  Training  School: 


Leo  Urbanek* 
Cheri  Moon* 

Ken  Heinlein* 


Jane  Taylor* 

Happy  Weston* 

Jack  Firestone 
Sharron  Kelsey 

Darryl  Cooper 

Jeff  Gardner 

Shelle  Glassman 

Ark  parents  and  consumers 


Jeanne  Thobro* 
Self-advocates  and  family  members 
Jayne  Mockler* 


David  Haines* 
Ramona  Doidge* 


Cheryl  Gerrish* 


John  Abas 


Terry  Cooper 
David  Heath 
Kathryn  Paxton 


Danny  Wilde* 


Shawn  Griffin* 
Bill  Davis 
Joe  Howard 
Pat  Harris 
John  Fontaine 

John  Holderegger 

Elva  Laird* 

John  L.  "Louie"  Gillette* 

Clarine  Law* 

Robert  T.  Clabby* 
Randy  Howdyshell* 
Hermann  Fischer* 
Linda  Gudmundsen* 
Becky  Nelson* 
Beverly  Swistowicz* 
Maribeth  Kendall* 
Cody  Kaper* 
Sue  Cloninger* 
Jim  Lyonn* 
Anita  Wilson* 


Brenda  Apadaca 
Jennifer  Johnson 
Sis  Nelson 
Gary  Hudson* 
Todd  Wright 


David  Brunner* 
Cliff  Mikesell* 
Charlotte  White* 
Becki  Harris* 
Patricia  Riddle* 
Denise  G.  Foss* 
Dawn  Parkins  Green* 
Tony  Martynuska* 
Mike  Auband* 
Mary  Lin  Johnson* 
Dennis  Yort* 
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Massachusetts 


Department  of  Mental  Retardation 


Phil  Campbell* 
Ron  Sanfield 
Terese  LaCroix 
Laurie  Anderson 
Mary  Ann  Brennan 
Connie  Lehr 
Rosemary  Bevins 
Jeanne  Adamo 
Jean  Tuller* 
Paul  Antonellis 
Hans  Toegel* 


Dorothy  Mullen 
June  Rowe* 
Gail  Grossman* 
Elliott  Berusch* 
Gerry  Morrissey 
Mary  Cerreto* 
Pam  Nicholson 
Michael  Kendrick* 
Dick  Cohen 
Angus  Capie 
Melissa  Probst 


Boston  Area  Advocates 
/Family  Members: 

Boston  Area  Providers: 


Jean  Whitney  Thomas 
Jerry  Kelly 
Lucy  Chansky 


Michael  Brooks 
Betsy  Clausen 
Jody  Williams 


Steven  Hoy,  Humanity  House 

Nancy  Hargreaves-Silver 

Diane  lagulli,  Delta  Project 

Todd  Kates,  UCPA  Metro  Boston 

Jennifer  Hight,  Bay  Cove  Human  Services 

Barry  Schwartz,  Vinfen 

Virginia  Connolly,  Beaverbrook  STEP 


Boston  Area  Consumers,  Family  and  Direct  Care  Staff 
TXIX  Surveyor 

Department  of  Public  Health:         Jack  Hart* 


Wrentham  Developmental 
Center  Staff: 


Wrentham  Developmental 
Center  QMRPs 


Southeast  DMR  Regional 
and  Area  Staff: 


Nicholas  D'Alusio 
Paul  Blake 
Cindy  Shephard 
Pat  Nichols 
Bruce  Ziegler 

Ann  Costa 
Paul  Maccaione 
Janet  Price 
Dean  Nowland 
Lady  Lavendar  Lee 

Rick  O'Meara 
Paul  Clifford 
Mark  Sanderson 


Dean  Nowland 
Joanne  Cumming 
Joe  Kavilhaug 
Mary  O'Leary 


Phil  Tully 
Randy  Webster 
Mary  Beth  Vargus 
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Southeast  Regional  Providers: 


Consumers: 


DMR  Western  Regional  Staff: 


Margie  Masse 
Rick  Cavicchi 
Marureen  Pereira 
Jack  Yates 


Dianne  Rodrigues 
Cecelia  Robers 
Kathleen  Catano 


Donna  Sabecky,  Behavioral  Associates  of  MA 

Peg  O'Keefe,  Community  Connections 

Jim  Ross  Community  Partnerships 

Janet  Butler,  Community  Systems,  Inc. 

Richard  McNally,  New  Bedford  Harbor  Services 

Ann  Shore,  CP.  of  South  Shore 

Suzanne  Raffa,  Reach,  Inc. 

William  Kelley,  Beta  Community  Service 

Karen  Seals,  Amego,  Inc. 

Steve  LeClair,  Randolph  Occupational 

Sheila  St.  Aubin,  Westport  Associates 

Paul  Hudson,  Kennedy-Donovan 

Susan  Harrington,  QUEST  Volunteer 
Mark  Wotjonski,  Future  QUEST  Volunteer 


Steven  F.  Bradley* 
Rick  Huntington* 
Lee  Schmidt* 
Doug  McAllen* 
Barbara  Morrel* 


William  Zimmer* 
Denise  Barci* 
Bruce  Brewer* 
Betsy  Noonan* 


QUEST  Surveyors:  Greg  Shumway* 

Neil  Lazar* 
Andrew  Fershette* 
Philiss  Seaver* 


Donna  Labello* 
Marty  Rachels* 
Bob  Sinopoli* 
Jim  Ciullo* 


Providers:  Ken  Singer,  Berkshire  County  ARC 

Janet  Ryan  Roman 

Terry  Blanchard,  Commonwealth  Community 
Services 

Barbara  Pilarcik,  Association  for  Community  Living 

Rich  Toscano,  Integra,  Inc. 

Ruby  Moore,  New  England  Business  Associates 

Families  and  Advocates:  Jim  Johnson,  Chair,  Local  Advisory  Board  and 

Parent 

Julie,  Region  I  Advisory  Board  -  Family  member 
served  by  Region  III 
Joyce,  Chicopee  Advisory  Board 
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Missouri 


Certification  Project  Team: 


Kate  McCiain 
Meg  Hutsier 
Gary  Davis 


Donna  Haley 
Cori  Brown 


University  of  Missouri,  Kansas  City  (UMKC-UAP) 
Also  present: 


Kansas  City  Regional  Center: 

St.  Louis  Regional  Center 

Sikeston  Regional  Center 
Allied  Grouos: 


Gail  Clair 
Charles  Wheeler 

Anita  Contreras 
Nancy  Shrewsbury 

Kim  Crites 

Carl  Calkins,  Director  UMKC-UAP 

Kay  Conklin,  Director,  Missouri  Developmental 

Disabilities  Planning  Council 
Bob  Fain,  Chair,  Missouri  Protection  and 

Advocacy  Board 

Shawn  de  Loyola,  Missouri  Protection  and 
Advocacy  Board 


Regional  Quality  Improvement  Staff: 
St.  Louis  Regional  Center 

Sikeston  Regional  Center 
Springfield  Regional  Center 


Also  present 


Certification  Steering 
Committee: 


DMRDD  Management  Staff: 


Anita  Contreras 
Nancy  Shrewsbury 

Kim  Crites 

Cindy  McKean 
Patty  Tapp 

Bob  Story*,  Parent,  Certification  Committee 
Kate  McCiain,  Quality  Enhancement  Coordinator 
Meg  Hutsier,  Certification  Consultant 
Gary  Davis,  Certification  Consultant 
Mary  Carrick,  Certification  Project  Committee 
Member 

Mary  Carrick 
Jean-Paul  Bovee 
Molly  Allen 
Bob  Story 

Melinda  Elmore 
Bob  Bauermeister* 
Bob  Thompson* 


8 


Also  present: 


Gail  Clair* 
Ray  Griffel* 
Jim  Magruder* 

Cori  Brown 
Donna  Haley 
Kate  McClain 
Crystal  Amini-Rad 


Family  Members  and  Advocates: 


Missouri  Advocates  for  Individuals   Jim  Spradling 
with  Developmental 
Disabilities  (MOAIDD)  Grandparent 


MOAIDD: 


Sherwood  Center: 
Also  present: 

ICF/MR  Administrators: 

Also  present: 


Core  Project  Management  Team 
on  Quality  Enhancement 
Strategies: 


George  Boyle 
Mike  Hanrahan 
Dick  Hosty 
Judy  Cross 
Marilyn  Spradling 

Marilyn  Stubbs 

Cori  Brown 
Donna  Haley 
Kate  McClain 

Bob  Bauermeister* 
Bob  Thompson* 
Ted  Allen* 

Cori  Brown 
Kate  McClain 

Denine  Hunt 
Ron  Tomer 


Core  Project  Management  Team  on  Quality  Enhancement  Strategies  (continued.) 

Ted  Allen 

Louise  Stefanowicz 
Molly  Allen 
Gail  Clairtive 


Also  present: 
Division  Director: 
Certified  Agencies: 


Crystal  Amini-Rad 
Greg  Kramer* 
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St.  Francois  County  Group  Home  Leanne 

Willows  Way  Kim 

Dawn 

ACD  Accredited  Agencies: 

Judevine  Center  Maura 

St.  Louis  ARC  Kathy  Meath 
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NEW  YORK 


Office  of  Mental  Retardation  and     Harold  Hopkins* 
Developmental  Disabilities  Staff    Thomas  Cuite* 

David  Picker* 
Tom  Richards 


Surveyors: 


PROVIDERS 
Albany  ARC: 

Rome  DDSO: 


Ron  Rzepnicki* 
Judy  Trent* 
Tom  Articola* 
Dan  Caryl* 


Larry  Fuld 
Andy  MacKenzie 
Deborah  Elcock 

Steve  Smits* 
Nancy  Straub* 
Sharon  Falke* 
Art  Holmberg* 
Sharon  Sherman' 


Mike  Savery* 
Joann  Caldara* 
Bob  Brooke* 
Sue  Brisley* 


George  Schlotterer* 
Diane  Dumas* 
Gil  Palladino* 
Gail  Rushford* 
Bill  Bonaro* 


AABR:  Peter  MaramakJi* 

Jane  Carnacchio* 
Kathryn  Flood* 
Doreen  Atwell* 


Saratoga  ARC:  Patrick  VanSlyke 

Consumers  and  Advocates:  Helen  DelVecchio 

Marisol  Diza 
Kermit  Augustine 
Marcie  Goldstein 
Rain  Rippel 

OTHER:  Bill  Combes,  New  York  State  Commission  on  the 

Quality  of  Care 

Sarah  Anderson,  New  York  State  Association  of 
Comunity  and  Residential  Agencies 
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ICF/MR  PROVIDERS  SURVIED 


Res-Care  Inc.  Helen  G.  FitzSimmons 
Houston,  Texas 

Independent  Options,  Inc.  Rebecca  Helgeson 
Corona,  CA 

Community  Support  Services,  Inc.  Rory  Robb 

Biddeford,  Maine  Susan  Church 

Ann  Miller 

Miami  Cerebral  Palsy  Marta  E.  Morlin 
Residential  Services,  Inc. 
Pembroke  Park,  Florida 

OHI  Bonnie-Jean  Brooks 
Hermon,  Maine 
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